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EXECUTIVE  SUMMARY 


During  the  contract  period,  Jonker  processed  10,000  technical  reports  for  the 
FAIRS,  increasing  the  size  of  the  central  collection  to  10,450  reports.  The  reports, 
covering  the  gamut  of  subjects  within  aviation  and  representing  the  technical 
information  that  host  describes  progress  in  that  field,  came  from  numerous  govern¬ 
ment  and  industrial  organizations,  but  the  largest  single  source  was  the  Federal 
Aviation  Agency  itself:  2, 202  were  generated  by  the  Agency  and  its  contractors. 
Jonker's  primary  objective  v'as  to  prepare  the  incoming  technical  information  for 
its  subsequent  retrieval  by  users  of  the  system.  In  brief,  Jonker  identified  the  im¬ 
portant  concepts  in  each  report,  indexed  these  concepts  with  suitable  keywords  from 
a  controlled  vocabulary,  and  entered  the  retrieval  information  into  the  system's  files 
and  searching  tools. 

Consistent  with  the  concept  of  centralized  processing-decentralized  operation, 
the  cost  of  Jonker's  work  was  incurred  only  at  the  Headquarters,  while  the  retrieval 
information  and  searching  tools  were  economically  duplicated  for  use  in  the  Regional 
Gifices,  the  Aeronautical  Center,  and  NAFEC.  The  Agency  had  created  these 
satellite  information  centers  to  serve  personnel  whose  remote  work  sites  might 
otherwise  leave  them  without  the  benefit  of  up-to-date  technical  report  information. 
Users  of  the  system,  either  at  the  Headquarters  or  the  field  installations,  can  com¬ 
municate  with  the  system  through  the  published  Thesaurus  of  FAA  Descriptors  which, 
as  the  system's  official  language,  links  them  to  indexers  who  put  reports  into  the 
system . 

Jonker  continued  development  of  the  Thesaurus  to  ensure  that  it  reflects  the 
latest  advances  and  terminology  of  aviation  and,  in  a  special  study,  measured  the 
influence  of  the  Thesaurus  and  of  indexing  and  searching  techniques  on  the  system's 
performance.  Data  were  acquired  to  tell  how  well  the  FAIRS  could  retrieve  docu¬ 
ments  that  were  relevant  to  users'  needs  and,  conversely,  how  well  it  could  avoid 
nonrelevant  documents.  The  system  responded  satisfactorily  to  a  variety  of  condi¬ 
tions,  but  revealed  sev  -al  areas  where  improvements  can  be  made. 

Specific  recommendations  are  offered  to  strengthen  report  acquisitions,  index¬ 
ing  and  shelving,  and  to  improve  the  Thesaurus  by  statistical  analysis  of  its  use. 
Other  recommendations  are  devoted  to  ways  of  increasing  the  users'  satisfaction 
with  the  FAIRS.  A  training-instructional  aid  could  broaden  the  usefulness  of  the 
system,  particularly  in  the  field  installations.  Improved  searching  techniques  also 
could  help.  Jonker  has  recommended  that  internally  developed  data  from  searching 
and  indexing  be  used  as  "feedback’’  in  guiding  the  future  growth  of  the  system. 

Jonker  ha?  recommended  that  special  attention  be  given  to  the  Improvements  that 
might  be  achieved  by  automatic  data  processing  techniques,  compatible  with  the 
punched  paper  taj*?  master  file  of  the  retrieval  information  that  Jonker  prepared. 
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INTRODUCTION 


This  final  report  summarizes  the  work  that  Jonker  Business  Machines, 
Inc.  has  accomplished  for  the  Federal  Aviation  Agency  under  contract  FA  64 
WA-5186:  Investigation  into  the  Retrieval  Indexing  and  Searching  System. 

The  subject  of  the  contractual  study  was  the  Federal  Aviation  Information 
Retrieval  System  (FAIRS),  which  serves  diverse  information  needs  of  the 
Agency's  technical  and  administration  personnel. 

Before  the  present  contract,  FAA  had  made  several  basic  decisions 
about  the  FAIRS  which  governed  Jonker 's  work.  For  one,  the  Agency  had 
decided  to  develop  a  controlled  vocabulary  especially  suited  for  civil  aviation, 
but  compatible  with  the  Thesaurus  of  ASTLA  Descriptors.  The  vocabulary, 
the  Thesaurus  of  FAA  Descriptors,  had  been  developed  under  an  earlier  con¬ 
tract  as  the  first  5, 000  reports  were  processed  for  the  system. 

Those  first  reports  were  implemented  into  the  search  and  retrieval 
tools  that  the  Agency  had  selected  for  the  FAIRS.  These  tools  Include  an 
identifier  file  for  special  nomenclature,  a  3  x  5  accession  card  file,  micro¬ 
film  copies  of  FAA  reports,  and  Termatrex  which  the  Agency  had  selected 
for  subject  searching. 

Prior  to  Jonker's  work,  moreover,  the  Agency  had  decided  to  exploit 
the  concept  of  centralized  processing-decentralized  operation  by  creating 
satellite  information  centers  in  its  field  installations.  The  Agency  chose  to 
bear  the  costs  of  report  analysis  and  processing  at  the  Headquarters,  but 
made  provisions  to  duplicate  the  search  and  retrieval  tools  for  each  Regional 
Office,  the  Aeronautical  Center  and  NAFEC. 

Another  decision  typified  the  Agency's  desire  that  the  report  process¬ 
ing  techniques  currently  used  in  the  FAIRS  be  compatible  with  automatic 
data  processing  techniques  that  might  be  used  in  the  future.  To  ensure  that 
compatibility,  FAA  required  that  Jonker  enter  the  information  needed  for 
retrieval  onto  punched  pa|>cr  tape  suitable  for  computer  input. 

Under  these  precedents  established  bv  the  Agency.  Jonker  helped 
develop  the  FAIRS  to  ope  rational  capacity  by  work  on  four  major  tasks  be¬ 
tween  July  1964  and  August  1965: 

1.  Assigning  descriptors  and  identifiers  to  reports  to  prepare  them 
for  subsequent  retrieval. 

2,  Expanding,  developing,  and  updating  the  system's  vocabulary: 
the  Thesaurus  of  FAA  Descriptors. 
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3.  Implementing  the  technical  reports  into  the  systom. 

4.  Investigating  tho  retrieval  effectiveness  of  the  system. 

Although  this  report  summarizes  all  the  work  undertaken  by  Jonker 
on  the  completed  contract,  it  concentrates  on  these  four  major  tasks .  The 
report  discusses  the  results  of  the  work  and  the  problems  that  were  en¬ 
countered,  and  it  provides  recommendations  for  future  procedures. 


WORK  ACCOMPLISHED 


Descriptors  and  Identifiers 

During  the  contractual  work,  10,000  technical  reports  were  prepared 
for  system  input  by  coordinate  indexing.  Ro|>orta  were  indexed  to  an 
average  depth  of  9.35  descriptors  each,  but  the  descriptor  assignments 
varied  from  2  to  25  per  report.* 

A  total  of  about  6, 280  Identifiers  were  also  assigned  to  the  reports. 
That  average  of  .  628  occurred  while  the  range  in  identifier  assignments  was 
from  0  to  13.  * 

The  Thesaurus 


The  Thesaurus  of  FAA  Descriptors  (1st  Ed.),  July  1964,  was  updated 
with  new  terminology  that  was  needed  to  adequately  index  the  reports:  356 
new  descriptors  were  added  to  the  Thesaurus,  but  176  established  descriptors 
were  deleted  for  lack  of  use.  The  net  increase  of  180  descriptors  raised  the 
total  now  in  die  vocabulary  from  2, 813  to  3, 093,  of  which  approximately  120 
are  not  in  the  A  ST  LA  Thesaurus  of  Descriptors.  The  "use"  references  in¬ 
creased  from  1, 848  to  1. 959. 

The  exhaustive  structures  and  cross-references  among  new  and  old 
terms  were  incorporated  into  a  final,  second  edition  of  the  Thesaurus. 
Camera-ready  copy  of  that  edition  was  delivered  at  the  completion  of  the 
contract.  Updated  EAM  cards,  the  storage  media  on  which  the  Thesaurus 
is  maintained,  also  were  prepared  and  delivered  to  FAA. 

System  Input 

The  10, 000  reports  indexed  were  entered  into  the  Termatrex  system 
which  had  been  installed  by  F.AA  for  searching  its  technical  report  literature. 
At  the  present  time,  two  Termatrex  card  decks  are  used  for  the  total  of 
16,  643  reports  in  the  collection. 

An  accession  record  file  was  maintained  for  each  of  the  technical 
reports  indexed.  FAA  was  able  to  provide  most  of  the  printed  3x5  abstract 
cards.  Because  the  cards  were  not  available  elsewhere,  catalog  cards  for 
4,696  of  the  reports,  well  over  FAA'a  contractual  estimate  that  ooe-third 
of  the  reports  would  need  cards. 


•  These  figures  are  comparable  to  those  reported  by  a  previous  contractor, 
where  the  average  indexing  depth  was  9.2  with  a  range  of  1  -  40;  and  the 
identifiers  averaged  0.95,  ranging  from  0  -  12. 
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Jonker  also  posted  accession  numbers  for  approximately  6, 280  of 
the  3  x  5  cards  in  the  Identifier  Card  file.  Of  these,  about  985  identifier 
cards  already  in  the  file  were  updated,  and  approximately  4, 000  new  identi 
fiers  were  added  to  the  file,  causing  its  size  to  grow  from  about  2, 700  to 
6,700  cards. 

The  10, 000  Document  Analysis  Worksheets  were  typed  as  (camera- 
ready  copy)  on  a  Flexowriter,  which  simultaneously  produced  a  punched 
paper  tape  master  file  containing  all  cataloging  and  Indexing  information. 
The  master  tapes  with  accompanying  correction  tapes  were  delivered  to 
FAA. 


Index  Files  for  Instruction  Manuals 

Index  files  on  3  x  5  cards  were  provided  for  4, 664  type  numbers 
from  instruction  books  and  maintenance  manuals  (not  technical  reports) 
that  were  located  In  the  FAA  Headquarters  Library  and  in  the 
referenoe  file  at  NAFEC.  Jonker  prepared  three  of  the  files,  one  each  for 
the  Headquarters,  Aeronautical  Center,  and  NAFEC.  Each  file  contained 
three  separate  decks  arranged  alphabetically  by  corporate  source,  alpha¬ 
betically  by  subject  heading,  and  numerically  by  type  numbers . 

Legal  Memoranda  and  Congressional  Materials 

The  review  and  collation  of  legal  memoranda  and  congressional 
materials  were  completed  in  November,  1964,  early  in  the  contract.  A 
separate  final  report  for  that  task,  submitted  at  that  time  as  specified  by 
the  contract,  has  been  included  as  Appendix  A  of  this  report. 

Retrieval  Effectiveness 


Studies  were  conducted  to  determine  the  system's  retrieval  effec¬ 
tiveness  according  to  how  well  it  avoided  non-relevant  documents  (Relevance 
Ratio)  and  how  well  it  retrieved  relevant  documents  (Recall  Ratio).  The 
performance  of  the  system,  measured  under  different  conditions,  was 
derived  from  the  test  data  as  follows: 

Relevance  Ratio:  35.4  -  59.3% 

Recall  Ratio:  22.2  -  73.3% 

Although  these  ratios  have  little  meantng  by  themselves,  it  will  be 
shown  later  that  the  effectiveness  tests  helped  explore  the  internal  work¬ 
ings  of  the  system:  indexing  techniques,  the  Thesaurus,  and  searching 
procedures. 
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DESCRIPTION 


General 


The  general  framework  for  the  technical  report  processing  used 
during  the  contractual  work  has  been  identified  within  the  Flow  Chart 
presented  in  Appendix  B  as  Figure  B-l .  Because  the  illustration  ties 
together  the  efforts  that  occurred  between  the  time  a  report  was  first  made 
available  for  processing  and  the  time  the  report  was  entered  into  the  sys¬ 
tem,  it  should  be  a  useful  reference  for  the  detailed  explanations  that  are 
presented  below. 

Document  Analysis  Worksheet 

The  heart  of  the  technical  report  processing  was  the  Document 
Analysis  Worksheet,  FAA  Form  3328,  shown  in  Figure  B-2.  The  DAW 
was  used  not  only  as  a  worksheet  for  indexing,  cataloging,  editing,  proof¬ 
reading,  and  TenWrex  drilling,  but  also  as  a  "source  document"  for  the 
Flexowriter  generation  of  punched  paper  tape  that  is  suitable  for  computer 
input. 


To  satisfy  the  ADP  requirement,  it  was  necessary  for  FAA  to  re¬ 
design  the  DAW  before  technical  report  processing  began.  The  size 
(number  of  digits)  needed  for  each  bibliographic  field  on  the  Worksheet 
was  determined  and  each  field  then  was  assigned  a  machine-readable 
identification  code  that  would  appear  in  the  paper  tap;.  Since  these  codes 
and  sizes  are  vital  tor  paper-tapc-to-magnetic-tape  conversion,  they  are 
presented  in  Figure  H-3.  The  form’  ?  margins  also  were  designed  so  that 
it  could  be  machine-fed  through  the  Flexowriter. 

Indexing 

Technical  reports  were  prepared  for  input  to  the  system  by  coor¬ 
dinate  indexing.  For  coordinate  indexing,  indexers  select  descriptors 
that  correspond  to  the  important  concepts  expressed  by  a  report.  Although 
the  descriptors  can  be  manipulated  individually,  they  can  be  used  in  con¬ 
junction  with  e;  ch  other  to  characterize  the  subject  matter  of  reports. 

At  FAA,  descriptors  (and  their  numerical  codes)  were  selected 
from  the  Thesaurus  of  FAA  Descriptors  (1st  Ed.),  July  1964,  by  experi¬ 
enced  indexers  who  have  had  nrofessionai  training  in  the  subjects  covered 
by  the  FAA  technical  report  collection.  To  promote  uniformity  among 


Indexers,  guidelines**  were  established  prior  to  the  start  of  indexing. 

The  guidelines,  coordinated  with  and  approved  by  the  FAA  Project  Officer, 
governed  the  depth  and  specificity  of  indexing  and  cited  reference  authori¬ 
ties  for  definitions  and  terminology. 

The  guidelines'  procedure  called  for  descriptors  to  be  entered  on 
the  DAW  at  tvo  levels:  primary  and  secondary.  Primary  terms,  designating 
the  subject  matter  of  gxe  atest  Importance  in  each  report,  were  preceded 
by  an  asterisk  on  the  DAW's. 

Indexers  also  entered  Identifiers  onto  the  Worksheets  to  cover 
specific  topics  in  the  reports,  such  as  airplane  models,  airport  names, 
equipment  designators,  helicopter  models,  project  names,  etc.,  that  were 
not  covered  by  descriptors. 

Each  completed  DAW  then  was  edited  to  provide  even  greater  con¬ 
sistency  over  the  indexers'  selections  of  descriptors  and  identifiers. 

Updating  the  Thesaurus 

Whenever  suitable  descriptors  could  not  be  located  in  the  Thesaurus 
of  FAA  Descriptors  (or  in  the  ASTIA  Thesaurus  of  Descriptors),  indexers 
establidi  ed  new  descriptors  for  inclusion  into  the  Thesaurus.  Each  sug¬ 
gested  vocabulary  change  was  transacted  via  a  Term  Justification  Form 
(TJF)  such  as  the  one  shown  in  Figure  B-4.  The  TJF  defined  each  new 
term  and  explained  how  it  would  affect  the  existing  Thesaurus  terminology 
by  portraying  the  new  tr  m's  structure  (cross-references  and  generic- 
specific  relationship).  TJF's  were  edited  by  the  Jonker  Project  Manager 
before  the  suggested  new  terms  were  submitted  to  the  FAA  Project  Officer 
for  approval. 

Approved  terms  and  their  structures  were  keypunched  and  merged 
into  the  Thesaurus,  which  has  been  maintained  on  EAM  cards;  rejected 
terms  were  given  lesser  status  as  identifiers.  Descriptors  which  had  beer, 
established  in  the  1st  Edition  of  the  Thesaurus,  but  which  were  not  used 
for  the  indexing  of  the  first  10,  000  reports,  were  deleted  from  the  2nd 
Edition  of  the  Thesaurus. 

To  further  assure  the  quality  of  the  revised  Thesaurus,  updated 
printouts  were  prepared  at  the  midpoint  and  the  end  of  indexing.  Each 
printout  was  carefully  edited  to  correct  any  weaknesses  in  the  descriptors' 
structures . 


*  Numbers  refer  to  references  presented  later  in  a  separate  section. 


Implementing  Reports  into  the  System 


Technical  reports  were  implemented  into  the  system  by  entering  the 
information  on  the  Worksheet  into  the  respective  files:  the  accession  reoord 
file,  the  identifier  file,  and  the  Termatrex  card  deck.  The  completed 
Worksheets  also  were  filed  after  they  had  been  typed  on  a  Programmatic 
Flexowritcr.  As  a  by-product  of  typing  the  DAW's,  the  Flexowriter  also 
produced  an  8-channel  punched  paper  tape  master  file  containing  the  reports’ 
complete  bibliographic  information  as  well  as  all  descriptors  and  identifiers. 

The  accession  record  file  was  expanded  by  serially  filing  a  catalog 
card  for  each  report.  Where  printed  cards  were  not  provided  with  the 
reports  by  FAA,  Jonker  produced  the  necessary  3x5  cards  (Figure  D-5) 
on  the  Flexowriter  by  recycling  a  second  by-product  paper  tape.  * 

The  identifier  file  was  updated  during  the  contract  period  by  posting 
report  accession  numbers  onto  applicable  3x5  Identifier  cards  such  as  the 
one  shown  in  Figure  B-6.  Additional  Identifier  cards  were  prepared  as  re¬ 
quired  . 

To  provide  the  FAIRS’  capability  for  subject  searching,  the  indexed 
reports  were  entered  into  the  Termatrex  system.  The  input  to  Termatrex 
was  made  via  an  automatic  drilling  device  (J-400)  that  read  verified  punched 
cards,  keypunched  from  proofread  Worksheets. 

As  the  deck  of  Termatrex  cards  for  the  FAA  Headquarters  was  drilled, 
additional  decks  were  drilled  for  each  of  the  seven  Regional  Offices,  and 
for  NAFEC  and  the  Aeronautical  Center.  By  allowing  a  reduced  cost  per 
deck,  the  simultaneous  input  for  Termatrex  cards  stands  as  a  major  exam¬ 
ple  of  the  economy  effected  by  the  concept  of  centralized  processing- 
decentralized  operation. 

During  Termatrex  card  drilling,  moreover,  the  mandatory  "generic 
posting"  of  index  terms  was  performed.  Generic  posting  requires  that  the 
card  for  a  generic  descriptor  be  posted  (drilled)  when  any  of  that  descriptor's 
specific  terms  have  been  selected  for  indexing.  As  an  example  taken  from 
the  Thesaurus,  CERAMIC  MATERIALS  (a  generic  term)  is  broader  than 
GLASS  (the  specific  term);  with  generic  posting,  CERAMIC  MATERIALS 
would  be  posted,  not  only  whenvor  it  was  used  for  indexing,  but  also  when¬ 
ever  GLASS  was  used  for  indexing.  (The  merits  of  generic  posting  are 
discussed  later  in  the  report.) 
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Preparing  Files  for  Manuals 


Three  index  files  for  maintenance  manuals  and  instruction  books  (one 
each  for  Headquarters,  for  NAFEC,  and  for  the  Aeronautical  Center)  were 
prepared  by  the  procedure  illustrated  in  Figure  B-7.  Jonker's  work  on  the 
task,  which  began  after  FAA  had  assigned  subject  headings  to  each  type 
number  in  the  manuals  and  books,  was  to  prepare  master  3x5  cards  that 
displayed  each  type  number  and  subject  heading  and  the  corporate  source 
which  produced  the  manual.  As  shown  in  the  example  of  Figure  B-8,  the 
title,  date,  and  contract  number  of  each  manual  were  added  to  the  master 
cards. 

The  first  group  of  manuals  that  were  processed  came  from  the  collec 
tion  at  the  Headquarters  Library .  The  finished  master  cards  for  them  then 
were  compared  to  the  collection  of  manuals  and  books  in  the  file  at  NAFEC, 
identifying  additional  manuals  for  processing.  During  the  comparison,  the 
file  location  of  each  manual  was  indicated  on  the  card,  so  that  the  completed 
cards  would  serve  as  a  consolidated  index  file  for  the  Agency. 

Jouker  printed  nine  copies  of  each  master  card  and  separated  them 
into  three  files,  each  containing  three  identical  decks  of  cards.  To  com¬ 
plete  each  file,  one  deck  was  arranged  in  order  alphabetically  by  corporate 
source,  one  numerically  by  type  number,  and  one  alphabetically  by  subject 
heading. 

Testing  the  System's  Effectiveness 

The  procedure  for  conducting  tests  of  the  system's  retrieval  effec¬ 
tiveness  is  explained  in  the  next  section  with  other  material  pertinent  to 
that  study . 


*  To  maintain  consistency  among  corporate  sources  for  these  cards,  Jonker 
catalogers  used  the  AEC's  Corporate  Author  Entries. 
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TESTING  THE  SYSTEM'S  RETRIEVAL  EFFECTIVENESS 


Description 

As  part  of  the  contractual  work,  the  FAIRS  was  tested  to  determine 
how  effectively  it  could  retrieve  reports  for  given  search  questions.  The 
test  was  designed  to  provide  information  on  the  effects  of  generic  posting, 
any  weakness  in  the  Thesaurus  as  a  searching  tool,  optimum  procedures  for 
searching,  and  the  adequacy  of  Indexing.  Although  the  complete  plan  for 
conducting  the  test  is  presented  in  Appendix  C.  pertinent  details  are  re¬ 
peated  in  this  section. 

In  brief,  the  entire  10,000-document  report  collection,  designated 
Document  Set  A,  was  searched  with  ten  test  questions.  The  questions, 
identified  in  Figure  D-13  were  selected  randomly  from  those  that  had  been 
previously  submitted  for  actual  searches  by  FAA  personnel .  *  All  available 
reference  questions  were  first  reviewed  and  separated  into  a  file  of  candi¬ 
date  test  questions .  A  question  was  not  considered  applicable  if  its  subject 
matter  was  no  longer  of  interest  or  value  to  the  respective  FAA  user.  A 
question  was  also  unacceptable  if  its  "author"  were  unwilling  to  participate 
in  the  test,  because  those  subject  specialists  were  later  «sked  to  judge  the 
reports  retrieved  by  the  system  for  their  relevance  (or  non-relevance)  to 
the  respective  question.  From  the  individual  users'  assessments  (Appendix 
D),  the  system's  performance  was  expressed  by  its  Relevance  Ratio  and  its 
Recall  Ratio . 

The  basic  ingredients  for  those  two  ratios  are  identified  in  the  2  x  2 
contingency  table  of  Figure  1.  From  the  notations,  the  Relevance  Ratio  can 
be  expressed  as: 

Relevance  Ratio  =  100  x  Reports  relevant  and  retrieved 

Total  Reports  Retrieved 

or  as  the  percentage  of  relevant  documents  retrieved  out  of  all  those  re¬ 
trieved.  (The  "non-relevance  ratio"  might  be  considered  to  represent  the 
noise  level  of  the  system's  retrieval.) 


*  It  was  considered  useful  to  re -impose  these  questions  on  the  system, 
because  the  report  collection  had  grown  by  over  50%  (from  6, 443  to  10.  000) 
since  the  original  searches  had  been  made. 
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Figure  1  -  INGREDIENTS  FOR  RELEVANCE  AND  RECALL  RATIOS 
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Recall  Ratio  =  100  x  Reports  relevant  and.retxieved 
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or  as  the  percentage  of  relevant  retrieved  out  of  all  thoce  reporta  which 
should  have  been  retrieved  (those  Judged  to  be  relevant  by  the  user) . 

To  derive  the  Reoall  Ratio,  It  became  necessary  to  determine  how 
many  reports  in  the  collection  were  relevant  to  eaoh  question:  not  only 
from  among  the  reports  retrieved,  but  also  from  among  those  not  retrieved. 
In  the  tests,  however,  the  entire  collection  was  not  assessed  for  eaoh 
question.  Instead,  the  assessments  were  made  from  within  a  10%  sample 
comprising  1,000  randomly  identified  reports  (Document  Set  B),  and  the 
results  from  searching  that  sample  were  then  extrapolated  to  the  collection. 

In  a  special  test  to  expand  die  data  for  the  Recall  Ratic,  moreover, 

20  questions  were  oomplled  from  the  texts  of  technical  reports  (Figure  D-14). 
These  "source  documents"  were  assumed  to  be  relevant  to  the  question 
that  they  inspired;  hence,  retrieval  from  these  20  questions  was  not  Judged: 
retrieval  was  successful  if  the  source  document  was  retrieved,  and  un¬ 
successful  if  the  source  document  was  missed. 

Even  though  the  system's  performance  could  be  expressed  in  these 
two  ratios ,  further  investigations  were  conducted  to  seek  the  causes  for 
the  system  1)  retrieving  non-relevant  material  and  2)  failing  to  retrieve 
relevant  material. 

As  a  first  step  in  isolating  the  causes  for  retrieval  failure,  the  test 
procedure  required  each  question  to  be  searched  by  four  different  strategies. 
All  four  strategies  required  the  use  of  the  Thesaurus,  but  they  differed  in 
the  level  of  descriptors  (generic  vs.  specific)  that  were  used. 

Strategy  A  was  composed  with  the  descriptors  e  ither  generic  or  speci¬ 
fic,  that  most  directly  corresponded  to  die  terminology  of  the  written  search 
question.  In  Strategy  B,  only  generic  terms  were  used.  The  general  intent 
of  this  strategy  was  to  retrieve  as  many  reports  as  possible .  Strategy  C 
used  the  same  generic  terms  that  were  used  In  Strategy  B,  but  the  non¬ 
pertinent  specific  descriptors  for  each  generic  term  were  substracted. 
Strategy  D  used  both  generic  and  specific  descriptors,  but  did  not  coordinate 
two  descriptors  of  the  same  level;  that  is,  the  applicable  generic  terms 
were  coordinated  only  with  the  applicible  specific  terms,  and  vice  versa. 
This  method  was  comparable  to  one  found  Bucoessful  at  E.  I.  duPont.2 

Relevance  and  Recall  Ratios 

The  retrieval  results  of  the  tests,  presented  in  Appendix  D,  have 
been  summarized  in  Tables  I  and  n.  The  data  of  Table  I  shows  that  the 
highest  Relevance  Ratio  for  the  system  was  59.3%,  achieved  with  Strategy 
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TABLE  I 

RETRIEVAL  DATA  FOR  RELEVANCE  RATIO 
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♦These  strategies  were  not  applicable.  * *Data  frost  Docncnt  Set  B  only 


RETRIEVAL  DATA  FOR  RECALL  RATIO 
(Document  Set  B) 


♦These  strategies  were  not  applicable. 


A;  the  poorest,  on  the  other  hand,  was  38.4%  with  Strategy  B. 

These  Relevance  Ratios,  however,  have  little  meaning  until  they  are 
coupled  with  their  respective  Recall  Ratios,  for  it  has  long  been  understood 
that  high  Relevance  is  attained  at  the  expense  of  Recall;  and,  conversely, 
that  high  Recall ,  at  the  expense  of  the  Relevance  Ratio .  The  data  of  Table 
n  supports  that  thesis:  the  highest  Recall  Ratio  (73. 3%)  of  Strategy  B  was 
achieved  with  the  lowest  Relevance  Ratio  in  the  test,  and  the  lowest  Recall 
Ratio  (22.2%)  accompanied  the  the  highest  Relevanoe  Ratio. 

Recall  data  from  searches  with  the  20  source  document  questions  de¬ 
picted  a  similar  pattern  among  the  four  search  strategies.  That  data,  presented 
in  Appendix  E,  are  summarized  in  Table  in  to  show  that  the  highest  Recall  Ratio 
of  90%  again  came  from  Strategy  B  and  the  lowest  (70%)  from  Strategy  A. 

TABLE  m 

RECALL  RATIOS  FROM  SOURCE  DOCUMENT  QUESTIONS 


Strategy  A 

70% 

Strategy  B 

90%. 

Strategy  C 

80%, 

Strategy  D 

75% 

Causes  of  Failure 

After  die  Relevance  and  Recall  Ratios  were  established,  an  investiga¬ 
tion  was  made  to  determine  the  causes  of  retrieval  failure  by  inspecting  each 
of  the  non -relevant  documents  retrieved  and  each  of  the  relevant  documents 
not  retrieved  during  the  effectiveness  test.  The  data  gathered  from  that  de¬ 
tailed  analysis, synthesised  into  later  discussions  about  features  of  the  system 
needing  attention,  has  been  summarized  in  Figure  2. 

The  two  major  causes  of  the  system's  failure  to  retrieve  relevant 
documents  were  in  searching  (43%)  and  in  Indexing  (38%),  and  the  retrieval 
of  non-relevant  documents  was  caused  primarily  by  searching  (33%)  and 
generic  posting  (44%). 

The  high  percentage  of  non-relevant  reports  retrieved  by  generic 
posting  appears  to  be  a  major  disadvantage  of  its  use  in  the  system,  but  it 
is  pointed  out  later  that  generic  posting  was  solely  responsible  for  the  suc¬ 
cessful  retrieval  of  61%  of  the  relevant  documents  retrieved  in  Strategy  B. 

The  failures  caused  by  indexing,  like  those  of  searching,  involve  sub¬ 
jective  intellectual  tasks  of  selecting  important  conoepts  and  their  applicable 
descriptors.  These,  as  others  have  found,  are  subject  to  error  in  any  system. 
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In  both  kinds  of  searching  errors,  the  critical  aspect  was  the  number 
of  search  terms  that  were  coordinated  simultaneously,  or  used  in  conjunction 
with  each  other.  The  coordination  of  three  of  four  terms  consistently 
yielded  few  reports  (including  few  relevant),  while  the  coordination  of  only 
two  terms  yielded  more  reports  (including  non-relevant  ones). 
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Comparison  of  FAIRS  to  Other  Systems 


Tho  Relevance  and  Recall  Ratios  determined  for  the  FAIRS  compare 
favorably  with  the  same  ratios,  shown  below  in  Figure  3,  developed  during 
tests  of  other  systems.  The  ratios  for  the  FAIRS  also  lie  within  the  general 
ranges  established  by  the  Cranfield  study: 

Relevance  Ratio:  1 0  -  25% 

Recall  Ratio:  60-90% 


Figure  3  -  TYPICAL  RELEVANCE  AND  RECALL  RATIOS 
{Reported  in  1964  Literature) 
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Navy,  BuShips 


|We stern  Reserve 
University 


Cranfield 


Air  University 


E.  I.  du'Pont 


K.  1.  duPont 


Documents 

in 

Collection 


1000 

1000 


(no  data) 


(no  data) 


6500 


(no  data) 


(no  data) 


Documents 

in 

Test 


1000 

1000 


950 


950 


6500 


433 


5000 


Defense  j  (no  data) 
Documentation 
Center 


000 


Relevance 

Ratio 


54.3% 

56.4% 


17.7% 


33.7? 


65.4% 


51-80? 


40-741 


76.7% 


*  Numbers  refer  to  References  in  a  later  section  of  this  report. 
*•  Determined  from  data  presented  in  Reference  2. 


Recall 

Ratio 


53.8% 


75.8% 


69.5% 


67-85% 


31-93% 


19  5% 


Despite  the  general  agreement  between  the  data  derived  from  the 
present  retrieval  tests  and  that  from  other  tests,  there  remains  debate 
among  documental ists  about  the  validity  of  mathematics  being  applied  to 
matters  as  subjective  as  "relevanoe."  No  correlation  between  a  user's 
real  need  and  his  expressed  need  (the  written  request)  has  been  made.  In 
one  of  the  test  questions,  for  example,  the  user  judged  eight  reports  rele¬ 
vant,  but  four  months  earlier  he  had  found  only  seven  of  the  same  documents 
relevant. 

Subjectivity  also  was  apparent  in  the  tests  when,  upon  Inspecting  the 
105  relevant  documents  not  retrieved  by  Strategy  A,  Jonker  indexers  (who 
are  professionally  trained  in  the  subject  matter)  disagreed  with  15  of  the 
assessments  that  had  been  made  by  the  FAA  staff  member.  Even  if  the 
users'  judgments  are  absolute ,  as  they  were  assumed  during  the  present 
tests,  it  remained  difficult  to  tell  with  certainty  always  where  the  system 
failed.  While  there  Is  no  clear  explanation  for  every  retrieval  failure,  few 
other  investigators  have  estimated  the  influence  of  subjectivity  on  the 
validity  of  their  test  data.  Some  have  even  acclaimed  "precise  values, " 
but  to  their  critics  dismay. 

Because  of  the  subjectivity  inherent  to  jo  many  facets  of  an  informa¬ 
tion  retrieval  system,  Jonker  suggests  that  the  performance  of  the  system, 
per  se,  has  not  been  establ  shed  even  though  the  Ratios  are  expressed  to 
three  significant  digits.  The  mathematics,  i.e. ,  the  ratios,  should  be  used 
only  in  context,  and  then  with  great  care.  In  the  present  tests,  moreover, 
the  data  is  susceptible  to  criticism  because  too  'ew  questions  were  used 
and  because  the  10%  sample  of  reports  (Document  Set  B)  did  not  produce 
consistent  results.  In  one  case,  for  example,  the  user  found  11  relevant 
reports  from  within  that  sample  and  found  another  11  from  the  rest  of  the 
collection.  Another  found  11  relevant  in  the  collection  but  only  1  in  the  sample. 

Jonker  believes,  nevertheless,  that  Relevanoe  and  Recall,  as  they 
were  used  in  the  present  tests,  can  he  used  to  detect  trends  within  an  in¬ 
formation  system  and  to  gauge  the  gross  (never  precise)  effects  of  the 
trends.  The  two  criteria  (and  the  analysis  of  system  failures)  highlight*! 
features  of  the  system  such  as  Uie  low  retrieval  caused  by  lightly  |x>sted 
descriptors,  the  lack  of  day-to-day  retrieval  data  caused  by  inadequate 
documentation  of  searches,  the  usefulness  of  "source  document”  questions 
to  an  indexing  supervisor  and  the  use  of  a  searching  technique  to  counter 
the  disadvantages  of  generic  posting.  The  test,  showing  its  merit  outside 
the  realm  of  Relevance  or  Recall,  helped  expose  the  problems  of  checking 
incoming  reports  for  duplicates,  of  maintaining  two  separate  shelf  loca¬ 
tions  for  reports,  and  of  shelving  reports  in  Corporate  Source  order. 

These  and  other  features  of  the  system  are  discussed  in  later  sections 
of  the  report. 


DISCUSSION 


Search  Strategy 

Taube  had  written  that  "most  librarians  and  information  people  know 
that,  in  general,  the  larger  tie(document)  3et  retrieved,  the  more  likely  it 
is  that  the  answer  will  contain  irrelevant  material  and  that  the  smaller  the 
set  retrieved,  the  more  likely  it  is  that  relevant  material  will  not  be  re¬ 
called/'7 

The  results  of  the  present  Relevance  and  Recall  tests  have  reiterated 
the  belief  that  differing  search  strategies  can  cause  the  system's  retrieval 
to  give  either  high  relevance  or  high  recall,  although  not  both.  What  is 
more  important  from  the  tests,  however,  is  that  they  have  shown  howto 
achieve  the  desired  variation  in  retrieval  results.  By  coordinating  three 
or  even  four  search  terms,  the  system's  retrieval  can  be  reduced  to  but  a 
few  documents.  That  strategy  would  be  applicable  where  a  user  wanted 
only  one  relevant  report.  If  only  the  most  generic  terms  are  coordinated 
as  they  were  in  Strategy  B,  the  number  of  reports  retrieved — relevant  and 
non-relevant — increases  greatly. 

A  compromise  between  the  very  specific  and  the  very  broad  waB 
afforded  with  Strategies  C  and  D.  Strategy  C  offered  a  slightly  higher  re¬ 
call  than  Strategy  B,  but  employed  a  cumbersome  technique  of  negations. 
Strategy  D.  on  the  other  hand,  involved  a  straight-forward  coordination  of 
specific  terms  with  generic  terms.  Because  this  Strategy  significantly 
reduced  the  number  of  non-relevant  documents  retrieved  without  seriously 
affecting  recall,  it  would  appear  that  Strategy  D  is  optimum  for  a  system 
that  uses  generic  posting.  Strategies  A  and  B  remain  applicable,  however, 
because  they  add  to  the  system's  ability  to  respond  to  the  users'  needs. 

To  exploit  this  ability  of  the  system,  the  searcher  must  know  whether 
a  specific  search  or  a  broad  search  is  desired,  and  should  also  know  how 
many  reports  the  user  wants.  (This  last  criterion  has  varied  a  great  deal 
among  users  of  the  FAIRS.) 

Search  Documentation 


From  the  review  of  reference  questions  prior  to  the  effectiveness 
tests,  -looker  was  able  to  detect  several  weaknesses  in  the  documentation 
that  is  presently  used  for  searching  in  the  FAIRS.  For  most  questions, 
the  search  records  (Figure  TV9)  lacked  notations  showing  which  technical 


reports  the  user  had  found  to  be  relevant  at  the  time  of  searching.  With 
no  identifying  accession  numbers  for  those  reports,  the  Search  Question 
Record,  FAA  Form  2712,  has  liftle  or  no  bibliographic  value:  if  the  same 
search  were  again  requested,  it  would  have  to  be  entirely  redone . 

Because  so  few  of  the  Records  identified  the  acoession  numbers  of 
the  relevant  and  non-relevant  reports  retrieved  for  these  actual  searches, 
it  was  impossible  for  Jonker  to  compare  the  results  of  Its  tests  to  the  day- 
to-day  satisfaction  that  users  have  with  the  system .  That  data,  if  provided 
on  the  Record  In  the  future,  would  easily  permit  the  Agency  to  compile 
massive  retrieval  data  similar  to  that  which  was  developed  during  Jonker's 
effectiveness  tests .  Such  a  continuing  survey  of  the  system  would  provide 
valuable  feedback  on  the  system's  retrieval  and  users,  as  well  as  data  that 
could  be  used  for  further  investigation  of  the  system. 

Documentation  for  searches  can  be  improved  in  other  ways  to 
s.rengthen  the  search  procedure  in  the  FAIRS.  For  one  imorovement,  the 
written  narrative  form  of  the  question  should  be  as  complete  as  possible. 
Many  of  the  Records  reviewed  by  Jonker  carried  only  the  searcher’s 
interpretation  of  the  question  (not  the  question  itself).  Interpretation  should 
be  closely  tied  to  use  of  the  Thesaurus .  As  a  good  exercise  to  encourage 
Thesaurus  use  before  searching,  the  Search  Record  should  provide  space 
for  writing  all  the  candidate  descriptors  which  might  be  used  for  searching. 

To  accomodate  the  suggested  additions  of  accession  numbers  and 
descriptors,  space  on  the  Record  could  be  alleviated  by  deleting  the  two 
sections  allotted  for  sub-questions.  The  suggested  revisions  in  the  Search 
Question  Record  are  shown  in  Figure  B-10. 

Generic  Posting 

Although  93,500  descriptors  were  posted  during  indexing,  Jonker 
estimates  that  generic  postings  have  increased  the  over-all  size  of  the  file 
by  about  50%;  that  increase  is  graphically  illustrated  in  Figure  4.  Since 
so  many  descriptors  (those  without  specific  references)  were  unaffected, 
however,  the  density  of  those  which  were  posted  increased  by  much  more 
than  50%.  As  shown  in  Table  IV,  many  postings  were  increased  by  well 
over  100%. 

The  effect  of  generic  posting  on  the  system's  retrieval  is  quite  ob¬ 
vious:  the  system  is  unable  to  yield  documents  indexed  solely  under  the 
generic  term.  Searching  with  generic  terms,  in  other  words,  retrieves 
unwanted  reports  from  he  species.  Of  the  non-relevant  reports  retrieved 
during  the  tests,  44%  were  retrieved  solely  by  generic  posting. 


20 


TABLE  IV 


POSTING  DENSITIES  OF  SELECTED  DESCRIPTORS 


Descriptor 

Posting  Density 

Without  generic 
posting 

With  generic 
posting 

AIR  ROUTE  TRAFFIC  CONTROL  CENTERS 

205 

J*. 

AIRPORT  CONTROL  TOWERS 

83 

♦ 

ATC  ENROUTE 

82 

* 

POSITIVE  CONTROL 

37 

* 

AIR  TRAFFIC  CONTROL  SYSTEMS 

889 

1023 

AIR  TRAFFIC  CONTROLLERS 

165 

* 

ATC  TERMINAL  AREA 

346 

* 

AERODYNAMIC  CONFIGURATIONS 

103 

961 

AIRCRAFT 

497 

2134 

AIRCRAFT  EQUIPMENT 

103 

481 

AIRPLANES 

35 

1154 

COMMUNICATION  SYSTEMS 

267 

742 

MECHANICAL  PROPERTIES 

141 

696 

PERSONNEL 

38 

497 

TERMINAL  FLIGHT  FACILITIES 

89 

824 

TEST  FACILITIES 

127  i 

i 

1 

253 

*  Generic  posting  not  applicable. 
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Despite  these  handicaps,  the  proponents  of  generic  posting  believe 
that,  in  general,  it  permits  a  system  to  attain  a  higher  Recall  Ratio,  knowingly 
at  the  expense  of  high  relevance.  The  data  from  the  present  tests,  in  general, 
support  the  use  of  generic  posting  in  the  FAIRS. 

\ 

For  one  thing,  the  tests  showed  that  the  "noise"  usually  associated 
with  generic  posting  can  be  reduced  by  a  technique  in  searching  (Strategy 
D»  where  coordination  of  one  generic  term  with  another  is  avoided.  In 
Strategy  P  the  Relevance  Ratio  was  44.9%,  the  second  highest  measured 
during  the  test.  In  Strategy  B,  on  the  other  hand,  generic  terms  were  co¬ 
ordinated,  and  the  Relevance  Ratio  was  the  lowest  (35.4%). 
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Although  the  Recall  Ratio  for  Strategy  D  (50.0%)  was  considerably 
lower  than  for  Strategy  B  (73.3%),  it  was  found  from  inspecting  documents 
that  the  Recall  Ratio  would  have  been  even  lower  in  all  strategies  had  generic 
posting  not  been  used.  Of  the  relevant  documents  retrieved  with  Strategy  B, 
61%  were  successful  solely  because  of  generic  posting  .'  (Generic  posting  also 
contributed  12  of  the  Source  Documents  with  Strategy  B.) 

The  argument  for  generic  posting  might  at  first  seem  to  apply  to 
smaller  collections  or  lightly  posted  terms,  but  the  ratio  of  Strategy  D  was 
effective  even  for  Question  270,  which  was  searched  with  two  descriptors 
(NOISE  and  AIRPORTS)  that  were  heavily  posted  both  from  indexing  and  from 
generic  posting. 

A  distinction  should  be  drawn  between  the  system's  Relevance  Ratio 
and  the  number  of  non-relevant  documents  it  retrieves .  The  Relevance 
Ratios  for  Strategies  A  and  D  were  both  near  50%,  but  Strategy  A  achieved 
that  ratio  while  retrieving  35  non-relevant;  Strategy  D  retrieved  131.  As 
the  system  continues  to  grow,  the  burden  of  these  non-relevant  documents 
(not  the  ratio)  could  become  excessive.  A  tolerable  response  can  be  achieved 
if,  as  was  pointed  out  earlier,  the  number  of  reDOrts  desired  by  a  user  is 
known  before  the  search  is  begun.  When  only  a  few  reports  are  wanted,  the 
search  can  be  narrowed  by  coordinating  three  or  four  descriptors  instead  of 
just  two.  Such  an  approach  is  needed  because  the  system  is  not  designed  to 
retrieve  small,  conveniently  sized  groups  of  reports  on  each  and  every 
subject:  the  system  has  many  reports  on,  say,  air  traffic  control,  and  it 
will  normally  yield  a  great  number  (both  relevant  and  non-relevant)  for 
searches  within  that  subject. 


Indexing 

From  the  Document  Analysis  Worksheets  inspected  in  conjunction 
with  the  Relevance  and  Recall  tests  and  from  data  such  as  that  presented 
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in  Table  IV,  Jonker  found  a  tendency  on  the  part  of  indexers  to  assign  the 
applicable  specific  terms  but  to  avoid  the  applicable  generic  terms.  General 
comments  from  indexers  confirmed  this  inclination,  and  they  attributed  It  to 
the  system's  mandatory  generic  posting  which,  they  were  subconsciously 
aware,  would  afix  the  generic  terms. 

The  combination  of  specific  terms  assigned  by  indexers  and  of  generic 
terms  automatically  posted  has,  nevertheless,  provided  an  adequate  re¬ 
trieval  file  for  the  FAIRS.  The  present  indexing  techniques  will  suffice  as 
long  as  the  "Post  Also"  feature  is  retained  in  the  FAIRS,  but  their  omission 
of  generic  terms  would  be  a  critical  weakness  if  generic  posting  is  discon¬ 
tinued. 


Steps  should  be  taken  now  to  assure  that  the  assignment  of  descriptors, 
either  generic  or  specific,  is  determined  by  the  document-at-hand.  As  an 
important  preliminary  change,  the  guidelines  for  indexers  should  be 
strengthened  to  emphasize  the  need  for  both  term-assignments.  The  tenden¬ 
cy  can  be  countered,  too,  by  the  editor -supervisor  who  reviews  the  indexers' 
selections  for  adherence  to  the  guidelines  before  reports  are  approved  as 
input  to  the  system. 

As  a  by-product  of  his  daily  editing,  the  supervisor  could  monitor 
descriptor  assignments  by  writing  hypothetical  search  questions  from  the 
tests  of  documents  just  indexed.  His  searching  could  be  made  (in  "longhand") 
from  the  Document  Analysis  Worksheets,  and  retrieval  success  or  failure 
would  provide  a  rough  score  for  indexers'  proficiency  (and  provide  them  an 
incentive). 

The  'proficiency  score"  of  indexers  in  this  case  assumes  the  same 
general  meaning  as  the  Recall  Ratio  as  it  was  developed  from  the  Source 
Document  Questions  searched  in  Jonker's  effectiveness  teat.  This  day-to- 
day  information  about  retrieval  would  be  valuable  feedback  for  the  system 
if  it  were  cumulated  monthly  and  coupled  with  the  Relevance  data  developed 
from  actual  searching. 

Because  indexers  must  anticipate  the  user's  viewpoint  and  relate  it 
to  the  document-at-hand,  it  was  helpful  during  the  contract  to  have  indexing 
and  searching  done  within  proximity  of  each  other  at  the  FAA  Library. 

Jonker  indexers  gained  even  more  insight  into  the  system's  uses  from  the 
analysis  of  Search  Question  Records  that  preceded  the  effectiveness  tests. 

The  analysis  of  that  collection  of  search  questions  also  identified  the  sub¬ 
ject  matter  most  often  requested  by  users  (the  ten  most  frequently  searched 
subject  areas  are  listed  in  Figure  5).  To  continue  these  benefits,  Jonker 
suggests  that  the  Search  Question  Records  be  reviewed  by  indexers  each 
month.  The  Records  should  l>e  reviewed  for  other  reasons,  as  is  suggested 
in  the  next  section  of  this  report. 
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Thesaurus  Adequacy 

With  the  addition  of  the  10,000  technical  reports  into  the  system,  a 
total  of  400  new  descriptors  were  suggested  for  addition  to  the  Thesaurus, 
and  356  of  them  (Appendix  E)  were  approved  by  the  FAA  Project  Offioer. 

The  new  terms  were  derived  primarily  from  the  input  side  of  the  system, 
having  come  from  report  indexing,  while  lesser  attention  was  given  to  the 
output  side  of  the  system.  New  terms  should,  nevertheless,  be  derived 
from  both  sides  of  the  system  so  that  the  Thesaurus  can  communicate  be¬ 
tween  the  indexers  who  put  reports  into  the  system  and  the  searchers  who 
take  reports  out  of  the  system .  The  search  questions  used  in  the  study  of 
the  system's  effectiveness,  for  example,  emphasized  the  value  of  several 
recently  established  descriptors,  such  as  AIRPORT  SURFACE  TRAFFIC 
and  CLEAR  AIR  TURBULENCE.  The  questions  also  provided  candidate 
descriptors,  such  as  BIRDS(INGESTION),  BIRD  STRIDE,  SCARE  DEVICES, 
and  LAND  USE . 

To  proximate  a  balance  between  current  literature  and  future  ques¬ 
tions,  changes  in  the  system's  vocabulary  should  reflect  both  those  influ¬ 
ences.  Because  the  users'  natural  terminology  is  expressed  on  the  Search 
Question  Records,  Jonker  believes  those  Record  t  should  be  reviewed 
monthly  by  the  persons  responsible  for  developing  the  Thesaurus . 

The  Thesaurus  growth  from  2, 813  descriptors  to  3, 093  descriptors 
(about  10^)  was  drastically  lower  than  the  almost-4-fold  increase  (from 
742  terms  to  2,  813  terms)  which  occvrred  from  indexing  the  system's 
first  5,000  reports.  Because  deletions  will  continue  to  accompany  addi¬ 
tions,  the  size  of  the  Thesaurus  could  stabilize  in  the  vicinity  of  3,000 
descriptors.  Deletions  will,  however,  more  critically  affect  the  future 
size  of  the  Thesaurus .  Some  statistics  about  descriptor  usage  are  valuable 
in  understanding  that  change. 

For  the  first  10,  000  reports  in  the  system,  indexers  used  descriptors 
an  average  of  29.5  times  each,  but  there  was  a  great  deal  of  variation 
among  their  posting  densities.  The  176  established  descriptors  that  were 
deleted  (Appendix  F)  had  never  been  used,  while  AIR  TRAFFIC  CONTROL 
SYSTEMS  was  used  for  889  reports.  (FEDERAL  AVIATION  AGENCY  was 
the  most-used  at  2,262.)  The  descriptors’  median  use,  a  more  significant 
statistic,  was  determined  to  be  only  seven.  As  shown  in  Figure  4,  the 
most  startling  value  is  the  mode  use  of  descriptors  at  only  one!  Although 
the  distribution  plotted  in  Figure  4  was  prepared  before  any  established  des¬ 
criptors  had  been  deleted ,  ii  can  readily  be  argued  that  the  Thesaurus 
still  contains  many  under-used  or  unnecessary  descriptors:  and  without 
generic  postings,  the  situation  would  lie  much  worse. 
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POSTINGS  PER  DESCRIPTOR 

Figure  4  -  DISTRIBUTION  OF  DESCRIPTOR  POSTING  DENSITIES 


As  the  system  grows,  some  of  these  descriptors  will  be  used  more 
frequently;  nevertheless,  each  should  be  evaluated  In  terms  of  Its  posting 
density  before  another  edition  of  the  Thesaurus  is  prepared.  Such  a  study 
of  descriptor  usage  should  be  conducted  to  correlate  the  relationships 
among  three  fundamental  variables  of  searching  and  retrieval  in  die  FAIRS: 
the  posting  densities,  the  number  of  search  terms  coordinated,  and  the 
number  of  reports  yielded.  The  benefits  anticipated  from  such  a  study 
would  be  in  terms  of  a  more  efficient  vocabulary  (fewer  terms  and  a  more 
economical  file)  and  of  a  more  satisfactory  search  procedure. 

Use  of  Termatrex 


By  analyzing  the  search  questions  which  were  imposed  on  the 
Termatrex  system  since  it  became  operational,  Jonker  has  been  able  to 
categorize  the  subjects  most  often  requested  (see  Figure  5) . 

Figure  5  -  SUBJECT  AREAS  MOST  FREQUENTLY  SEARCHED 

IN  THE  FAIRS 


AIR  TRAFFIC  CONTROL 

AIRCRAFT  NOISE  AND  SONIC  BOOMS 

AIRPORTS 

ELECTRONICS 

FAA  POLICY 

HAZARDS  AND  SAFETY 

METEOROLOGICAL  PHENOMENA 

NAVIGATIONAL  AND  LANDING  AIDS 

RADAR 

SUPERSONIC  TRANSPORT 


As  in  any  information  svstem,  the  report  acquisitions  (and  report  indexing) 
should  reflect  the  users'  interests  that  are  expressed  in  these  10  categories. 

From  the  same  analysis  it  was  found  that;  during  the  last  year,  24 
questions  from  within  the  Headcpiarters  were  applicable  to  Termatrex 
searching  although  Termatrex  was  not  used;  a  similar  problem  surely  exists 
in  the  Regions.  Some  of  these  omissions  can  be  attributed  to  the  newness 
of  the  FAIRS,  so  they  should  be  less  frequent  in  the  future.  The  Agency 
has,  moreover,  already  devoted  some  attention  to  the  training  of  personnel 
who  should  use  the  system. 
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Because  of  personnel  turnover  (If  for  no  other  reason),  such  training 
is  a  continuing  necessity  for  the  FAIRS.  The  largest  possible  population  of 
users  should  be  aware  of  the  kinds  of  information  in  the  collection  and  the 
retrieval  tools  available  to  them .  The  training  function  applies  to  all  FAIRS 
Installations ,  but  the  lack  of  it  would  be  particularly  acute  in  the  field 
locations,  where  the  system  is  nearly  a  year  newer  than  it  is  at  the  Head¬ 
quarters. 

As  a  first  step  in  providing  user  training,  die  Agency  should  consider 
a  basic  operating  manual  that  would  serve  the  system's  operators,  too. 

Less  passive  aids,  such  as  standard  briefings,  training  films,  special 
slides,  etc.,  should  also  be  considered. 

Report  Shelving 

One  by-product  of  Jonker's  indexing  and  effectiveness  tests  was  the 
inadvertent  discovery  of  reports  that  were  already  in  the  collection.  Al¬ 
though  reports  to  be  indexed  were  selected  by  the  Information  Retrieval 
Branch,  HQ-600,  it  remained  possible  for  Jonker  indexers  to  recognize  45 
duplicate  reports  (even  when  reviewed  days  apart)  and  to  return  them  to  the 
Project  Officer  before  they  had  been  reprocessed. 

Still  other  duplicates  were  not  detected  untL  after  they  had  already 
been  entered  into  the  system.  Of  the  10, 000  reports  indexed,  84  were 
found  to  be  duplicates  after  they  had  been  reprocessed.  That  number  is  a 
relatively  low  percentage  of  the  total  processed,  but  it  represents  a  sizea¬ 
ble  processing  cost. 

Checking  incoming  materials  for  duplication  is  an  essential  step  in 
any  information  system  and,  quite  properly,  materials  at  FA  A  are  being 
checked.  The  tools  available  for  finding  duplicate,  however,  are  far  from 
adequate.  In  many  cases,  the  incoming  document  must  be  physically  com¬ 
pared  to  the  ones  already  shelved,  which  is  more  cumbersome  and  time- 
consuming  than  comparisons  made  within  some  form  of  report  index. 

More  importantly,  because  the  reports  are  shelved  in  corporate- 
source  order  for  duplicate  checking,  they  are  difficult  to  retrieve  and  file 
for  day-to-day  searches.  It  is  estimated  that  half  of  the  time  spent  on 
filing  and  retrieving  might  be  saved  if  serial  number  (accession  number) 
order  were  used  for  shelving.  The  user's  time  is  also  spent  while  he  waits 
for  reports  to  be  pulled.  The  conversion  of  the  shelves  from  corporate 
source  to  accession  number  order  must  await  the  development  of  an  ade¬ 
quate  tool  to  replaoe  the  existing  technique  of  duplicate  checking,  i>uch  as 
a  book  or  card  catalog  organized  as  an  index  of  report  title .  The  catalog 
could  be  prepared  as  a  computer  printout  from  the  data  now  available  in 
punched  paper  tape. 
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Medical  Reports 


The  acquisition  of  medical  and  non-medical  reports  should  be  better 
coordinated.  Medical  reports  arq  selected  for  the  system  on  the  basis  of 
their  subject  matter  by  the  Medical  Librarian,  while  other  reports  are 
selected  by  personnel  in  the  Information  Retrieval  Branch.  Thus,  when 
the  subject  matter  is  not  distinct  (say  in  Bionics  or  Physiology},  a  candidate 
report  could  be  selected  by  both  groups  or  by  neither.  To  correct  this 
procedure,  the  two  groups  should  coordinate  the  materials  that  they  scan 
for  report  acquisition . 

Book  or  card  catalogs  would  have  value  as  retrieval  tools  for  the 
FAIRS,  which  now  has  only  a  subject  searching  tool  (Termatrex).  Their 
value  would  be  particularly  applicable  to  medical  reports,  where  the 
author's  name  is  often  as  important  for  retrieval  as  the  descriptors  for 
his  subject.  This  trait  of  (nedical  literature  partially  explains  why  so  few 
medical  searches  were  found  among  the  reference  questions  that  were  re¬ 
viewed  prior  to  Jonker's  effectiveness  tests.  None  of  die  ten  questions  used 
in  the  test  were  on  medical  subjects . 

As  another  weakness  of  the  FAIRS,  medical  reports  are  snelved  in  an 
area  physically  separate  from  die  other  reports.  The  system  bears  the 
common  burdens  of  two  different  files,  but  because  the  accession  cards  are 
not  marked  to  designate  medical  reports,  the  user  (or  FAA  staff  member) 
is  not  directed  to  the  appropriate  file.  This  Inequity  could  be  eliminated 
by  simply  adding  such  a  code  to  the  accession  cards. 

Identifier  File  Size 


The  idertlfier  file,  now  at  about  6,700  cards,  was  increased  by  4,000 
cards  during  the  project,  or  about  .40  cards  per  report.  That  growth  rate 
is  only  slightly  less  than  the  rate  of  .54  per  report  which  accompanied  the 
first  5,000  reports  in  the  system.  If  not  regulated,  the  file's  size  will 
soon  become  a  problem.  (By  the  time  another  10,  000  reports  are  in  the 
system,  it  could  have  exceeded  10,000  cards.)  Some  further  explanations 
should  be  useful,  however,  before  a  means  is  chosen  to  control  that  file. 

Because  of  its  nature,  the  identifier  file  can  logically  be  expected  to 
continue  growing  faster  than  the  Thesaurus.  For  example,  the  Thesaurus 
contains  about  52  descriptors  to  cover  kinds  of  aircraft  (including  the  des¬ 
criptor  AIRCRAFT),  but  it  will  acquire  additional  descriptors  only  as  new  ! 

kinds  of  aircraft  become  known.  The  identifier  file,  on  the  other  hand,  j 

includes  hundreds  of  cards  for  the  names  of  the  aircraft  and  can  be  ex-  j 

paneled  with  the  names  of  each  new  model  that  is  developed.  Because  these  j 
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aircraft  models  become  obsolete  or  are  renamed,  moreover,  the  identifier 
file  always  contains  information  of  temporary  value.  In  short,  the  file  is 
characterized  by  continual  growth  and  continual  obsolescence. 

Two  basic  ‘•/ays  of  overcoming  those  traits  of  die  file  might  be  used 
at  FAA.  The  first  method  is  to  weed  out  old,  existing  identifiers.  The 
entire  file  could  be  reviewed,  say,  annually  by  assessing  the  value  of  each 
identifier .  Because  proportionately  few  of  the  existing  identifers  are  now 
old  enough  to  be  weeded,  however,  this  method  would  be  more  applicable 
in  the  future.  The  seoond  basic  method  is  to  restrict  the  number  of  identi¬ 
fiers  presently  going  into  the  file.  Identifier  selection  could  be  limited  to 
important  classes  such  as  airplane  engines,  airplane  models,  airport 
names,  helicopter  models,  FAA  project  names,  etc.  The  desired  classes 
could  be  specified  by  a  revision  in  the  guidelines  for  indexers.  Of  the  two 
methods,  the  latter  is  preferable  because  it  is  applicable  now  and  because 
it  requires  lees  effort  to  achieve  the  same  result. 

Personnel 


About  12,  81?  man-hours  were  spent  on  the  project  and  were  allocated 
to  various  tasks  as  shown  in  Table  V. 


Table  V 

ESTIMATED  TIME  PER  PROJECT  TASK 


TASK 

MAN-HOURS 

Professional 

Total 

Processing  Reports 

Indexing 

2.651 

2,651 

Cataloging  &  Coding 

l  .495 

372 

1,867 

Editing 

930 

930 

Flexowriter 

1,422 

1,422 

Implementation 

529 

529 

Developing  Thesaurus 

•Justifying  New  Terms 

241 

241 

Editing  Structures 

11s 

27  U 

394 

Preparing  Index  for  Manuals 

1 . 652 

26 

i  ,679 

Reviewing  Legal  Memoranda 

$62 

6!  2 

1 . 274 

Testing  the  System's  Retrieval 

140 

423 

563 

Training 

232 

393 

625 

| 

Project  Planning 

1  0 

629 

639 

TOTAL: 

6.  260 

6,553 

12,81  3 

The  processing  rates  for  two  key  tasks,  depicted  in  Figure  6,  show 
that  the  project  started  slowly  but  finished  nearly  on  schedule.  Early 
slippage  on  the  project  was  caused  by  the  time  devoted  to  training  and 
by  delays  in  preparation  of  the  intricate  program  tapes  for  die  Flexowriter 
processes . 
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REPORT  PROCESSING  RATE  DURING  THE  PROJECT 


CONCLUSIONS 

It  is  concluded  that: 

1.  The  design  and  use  of  the  Search  Question  Record  is  inadequate. 
The  Record  would  have  more  value  in  bibliographic  reference  if  it  displayed 
a  full  narrative  form  of  each  question  and  the  accession  number  of  reports 
retrieved  for  it.  To  encourage  use  of  the  Thesaurus  before  searching,  the 
record  also  should  display  the  candidate  descriptors  that  might  be  used  for 
each  question. 

2.  The  future  growth  of  the  FAIRS  oan  be  guided  by  two  kinds  of 
"feedback  information"  derived  from  the  system's  present  operation: 

a .  Relevance  data  (denoting  user  satisfaction)  as  it 
is  recorded  on  the  Search  Question  Records. 

b.  Recall  data  acquired  via  "Source  Document"  ques¬ 
tions  posed  by  the  indexers'  supervisor. 

3.  Precautions  are  needed  to  ensure  that  the  indexers'  selection 
of  generic  descriptors  is  governed  by  the  document-at-hand  and  not  by  the 
system's  automatic  generic  posting.  Their  tendency  to  emphasize  the 
specific  descriptors  can  be  countered  by  training  (via  die  Guidelines  for 
Indexers)  and  by  the  source  document  questions . 

4.  The  future  size  of  the  identifier  file  should  be  limited  by  con¬ 
trolling  what  goes  into  the  file .  Weeding  the  existing  identifiers  is  an  im¬ 
practical  way  to  reduce  that  file  at  this  time . 

5.  A  new  information  system  such  as  the  FAIRS  should  provide 

a  means  for  training  its  users  (and  operators)  so  that  they  are  familiar  with 
the  information  tools  of  the  system  and  with  the  techniques  of  retrieval. 

This  function,  especially  important  to  the  field  installations,  can  be  satis¬ 
fied  by  a  training- instructional  aid  such  as  an  operating  manual,  training 
film,  etc. 

6.  Generic  posting  enhances  the  system's  ability  to  retrieve 
relevant  reports.  The  irrelevant  reports  it  also  retrieves  can  be  reduced 
through  a  search  strategy  wherein  the  coordination  of  two  generic  des¬ 
criptors  is  avoided. 

7.  The  number  of  seldomly  used  descriptors  in  the  Thesaurus 
poses  a  problem  requiring  additional  study.  Those  descriptors,  an  un¬ 
economical  segmeni  of  the  vocabulary,  make  the  search  procedure  cum¬ 
bersome  . 
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8.  The  workload  and  Inaccuracy  of  acquis  Itioning  new  reports 
would  be  lessened  if  the  incoming  material  was  compared  to  an  index  instead 
of  to  the  shelf.  A  book  catalog,  now  feasible  with  the  punched  paper  tape  as 
computer  input,  should  be  organized  by  report  title .  Alternate  arrangements 
by  corporate  source  or  personal  author,  although  more  prone  to  error,  can 
be  considered. 

9.  Shelving  reports  in  accession  number  order  would  improve  the 
general  operation  of  the  system,  making  it  more  responsive  to  day-to-day 
searching.  The  conversion  of  the  shelves  from  corporate  source  order  to 
accession  number  order  must  await  development  of  a  suitable  tool  for  dupli¬ 
cate  checking. 
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HKCOMMENDATIONS 


It  is  recommended  that: 

1.  The  Search  Question  Record,  to  be  used  for  all  search  questions, 
be  re-dcstgned  as  suggested  in  Figure  B-10  to  provide  more  space  for  the 
following: 

a.  The  narrative  form  of  each  question. 

b.  The  candidate  descriptors  for  searching. 

c.  The  accession  numbers  of  reports  that  were  found 
to  satisfy  the  user. 

2.  "Source  Document"  questions  be  written  daily  by  the  editor- 
supervisor  to  check  indexers'  proficiency  and  to  acquire  feedback  informa¬ 
tion  about  the  system's  retrieval. 

3 .  The  Guidelines  for  Indexers  be  revised  to  encourage  indexers  to 
equally  emphasize  specific  and  generic  descriptors  and  to  restrict  their 
selection  of  identifiers  to  only  the  most  im|>ortant  classes. 

4.  The  Agency  undertake  development  of  a  training-instructional 
aid  for  users  of  the  system  at  the  Headquarters  and  field  installations.  (An 
operating  manual  should  be  considered  a  minimum.) 

5 .  Generic  posting  be  continued  at  the  present  time .  As  a  distant 
future  consideration,  before  the  index  file  is  converted  to  automatic  data 
processing,  the  economics  of  generic  posting  should  be  reconsidered. 

6.  A  special  study  of  descriptor  usage  be  conducted  to  find  an 
optimum  vocabulary  size  and  more  economical  search  procedures  for  the 
system . 


7.  A  cost-benefit  study  be  conducted  to  determine  the  economic 
feasibility  of  using  ADP  techniques  to  prepare  report  catalogs,  with  the 
available  punched  paper  tape  as  computer  input.  The  arrangements  to 
consider  for  the  catalogs  are: 

a.  By  title,  for  duplicate  checking, 

b.  By  author  for  retrieval, 

c.  By  corporate  source,  for  retrieval. 

8.  Contingent  upon  the  development  of  the  title  catalog,  the  re¬ 
port  shelving  be  converted  from  corporate  source  order  to  accession  number 
order . 
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9.  To  avoid  inadvertently  adding  duplicate  reports  to  the  system, 
the  scanning  of  announcements  materials  be  coordinated  between  the  Medical 
Library  and  the  Information  Retrieval  Branch. 

10.  The  accession  cards  for  medical  reports  l>e  marked  to  reveal 
the  special  file  location  of  those  reports. 
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FOREWORD 


This  report  summarizes  the  work  accomplished  under  Article  I,  Items 
4  and  5  of  Contract  FA  64WA-5186,  which  required  die  processing  of  legal 
memoranda  and  Congressional  materials.  The  report  presents  some 
statistics  related  to  the  work,  and  discusses  some  of  the  problems  en¬ 
countered  during  the  effort.  The  report  also  recommends  procedures  to 
be  followed  in  the  future . 
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I.  INTRODUCTION 


In  close  coordination  with  the  Law  Library,  personnel 
from  Jonker  Business  Machines,  Inc.  performed  several  taska 
related  to  Legal  Memoranda  and  Congressional  Materials.  The 
■ajor  tasks  Involved  were  as  follows: 

Legal  Metsoranda 

Reviewing  of  selected  memoranda 
Charging  and  dlacharglng  materials  frost  the  file 
Collating  with  memoranda  froa  other  collections 
Listing  memoranda  by  subject 

Congressional  Materials 

Collating  by  congress  end  type  of  material 
Preparing  aaterlals  for  binding 
Reviewing  legislative  histories 
Acquiring  missing  aaterlals 


II.  LEGAL  MEMORANDA 


Work  Accomplished 

As  the  first  step  toward  eoaplllng  a  complete  set  of 
legal  memoranda,  memoranda  froa  35  file  drawers  of  General 
Files  were  reviewed  according  to  the  following  criteria: 

1.  Having  permanent,  historical,  or  lasting  value. 

2.  Containing  legal  precedent. 

3.  Containing  legal  research. 

During  this  review,  1,721  folders  containing  a  total 
of  over  100,000  pages  were  processed  and  3,211  memoranda  were 
retained.  The  3,211  Items,  averaging  about  two  pages  In 
length,  were  temporarily  charged  from  the  file  systems,  micro¬ 
filmed,  and  reproduced  to  facilitate  their  comparison  with 
memoranda  from  three  other  collections. 

The  other  collections  contained  an  additional  4,800 
Items  of  which  about  1,300  were  chronologically  merged  Into 
the  first  group  to  give  a  complete  set  of  legal  memoranda 
comprising  about  4,500  Items. 

Accession  numbers  had  been  assigned  to  the  selected 
memoranda  and  listed  adjacent  to  their  respective  file¬ 
system  subject  for  cross-referencing  a  search  of  the  memoranda. 
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Time  Spent 

The  reviewing  of  memoranda  required  208  houre  by  evo 
profeeelonel  eteff  members  who  hed  subject  background  in  lev 
(one  was  a  lew  school  graduate).  These  professional  personnel 
and  a  clerical  person  spent  an  additional  182  hours  to  charge 
and  84  hours  to  discharge  the  memoranda  from  the  files;  they 
spent  another  299  hours  collating  the  collections  of  memoranda. 
The  number  list  for  cross-referencing  required  32  hours. 

Coordination  of  the  microfilming  and  reproduction  pro¬ 
cesses  with  other  work  absorbed  another  90  hours. 

Problems  Encountered 


No  serious  problems  were  encountered  during  this  phase 
of  the  project;  however,  two  minor  problems  were  encountered. 
One  involved  the  tedium  in  charging  and  discharging  memoranda 
from  the  files  and  in  collating  the  8,000  items  of  the  four 
collections.  The  details  of  these  particular  tasks  required 
unusual  attentiveness  for  several  consecutive  workdays.  In  the 
future  such  tasks  should  be  Interrupted  by  a  “change  of  pace;" 
such  a  convenience  was  not  possible  on  this  project. 

The  second  problem  involved  the  microfilming  and  subse¬ 
quent  reproduction  of  memoranda:  swre  time  than  anticipated 
was  spent  on  the  coordination  of  these.  That  problem  is  sosw- 
vhat  unique  to  this  project  In  that  unusually  critical  dead¬ 
lines  had  to  be  met.  In  one  case,  for  example,  the  memoranda 
could  not  be  available  until  a  Friday  night,  but  had  to  be 
reproduced  and  returned  by  Monday  morning. 


III.  CONGRESSIONAL  MATERIALS 


Work  Accomplished 

Congressional  materials  from  the  74th  Congress  through 
the  88th  Congress  were  collated  by  Congress  and  by  type  of 
material!  Hearings,  Reports,  Documents ,  and  Prints.  Within 
each  type,  the  materials  were  arranged  by  their  source  (Bouse, 
Senate,  or  Joint)  and  alphabetised  by  title.  Theae  materials 
were  a<. tanged  in  ahelvea  where  130  ltesu  were  marked  for  bind¬ 
ing;  130  legislative  histories  were  reviewed  and  missing  items 
that  wars  available  elaeidtere  within  the  Federal  Aviation 
Agency  were  acquired. 

Duplicate  copies  of  these  materials  were  similarly  ar¬ 
ranged  on  the  library  shelves. 
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Ttas  Sptnt 


A  total  of  379  man-hours  viri  spent  arranging  congressional 
materials.  About  300  houra  vara  apant  actually  collating 
materlalsj  tha  remainder  waa  apant  preparing  materials  for  tha 
binder,  shelving  duplicate,  reviewing  leglalatlva  hlatorlaa, 
and  acquiring  alaelng  ltaaa. 

Problems  Encountered 


The  only  notable  problem  encountered  with  the  eongreaalonal 
•atarlala  waa  that  they  could  be  collated  only  by  a  person 
faalllar  with  their  types.  That  person  had  to  recogulze  the 
obscure  differences  aaong,  for  -  staple ,  Reports,  Docuaenta,  and 
Prints . 

The  specialized  nature  of  these  congressional  materials 
restricted  the  asslgnaent  of  other  personnel  to  that  task  al¬ 
though  their  tlae  was  occasionally  available. 


IV.  RFCOMENDATIONS 


As  a  result  of  the  work  done  on  Iteas  4  and  5  of  Contract 
FA  64WA-5786,  a  aeries  of  general  reconendatlona  can  be  aade 
to  aaterlally  assist  the  General  Counsel’s  Office  of  FAA  In  the 
retrieval  of  legal  documentation  necessary  to  their  work. 

At  this  point  In  time  and  with  little  knowledge  of  user 
requirements,  however,  the  whole  panorama  of  an  Information 
Retrieval  System  and  Its  lntra-relatlonshlps  with  the  communi¬ 
cation  language  and  retrieval  hardvare  cannot  be  specified  in 
great  detail.  Nor  can  one  state  precisely,  now,  the  type  and 
depth  of  Indexing  (Intellectual  analysis  of  substance  of  the 
documents)  required  for  legal  documents.  All  of  these  variables 
lnter-act  with  one  another  and  the  realities  of  FAA's  economics. 

In  spite  of  the  restrictions  mentioned  above,  It  la 
recommended  that  the  Federal  Aviation  Agency  pursue  an  Informa¬ 
tion  system  for  legs]  Information  chat  embodies  the  concept  of 
centrallzed-processlng  but  decentrallzed-operatlon,  comparable 
to  how  the  concept  Is  embodied  In  the  Federal  Aviation 
Information  Retrieval  Svstem  (FAIRS)  now  operating  at  FAA  Head¬ 
quarters. 

As  the  first  step  toward  implementing  that  concept.  It 
la  recosamended  that  a  "feasibility  experiment"  be  conducted  to 
explore  such  variables  as:  user  requirements,  language  prob- 
leas  Inherent  in  the  system,  indexing  problems,  and  the  cost 
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trade-offs  possible.  Such  an  experlnent  could  be  undertaken 
either  by  FAA  or  by  an  outside  contractor  (In  either  case, 

FAA  would  be  heavily  Involved),  but  should  Include  and  follow 
the  protocol  outlined  below: 

1.  A  sufficient  sample  of  existing  FAA  legal  opinions 
and  legal  memoranda  should  be  Indexed  by  their  sub¬ 
stantive  retrieval  concepts,  utilizing  accepted  legal 
terms  and  additional  factual  terms  as  applicable,  to 
determine  generalized  requirements  of  a  thesaurus  of 
legal  terminology  of  FAA. 

a.  The  sample  should  be  materials  from  latter  years, 
chosen  at  least  partially  from  those  among  the 
memoranda  chronologically  collated  during  the 
task  Just  completed. 

b.  This  sample  should  be  compatible  with  Regional 
-  needs. 

c.  The  sample  should  include  material  topics  of 
Procurement  Law,  general  legal  services,  litiga¬ 
tion,  airports,  regulations,  and  enforcement. 

d.  The  sire  of  the  sample  will,  of  course,  be  e 
direct  function  cf  economics. 

2.  The  thesaurus  of  terms  generated  from  indexing  opin¬ 
ions  and  memoranda  should  be  extensively  edited  and 
criticized  by  FAA  staff  attorneys  at  Headquarters  and 
at  Regional  Offices,  who  have  practical  knowledge  of 
legal  documentation  and  the  requirements  to  be  im¬ 
posed  upon  its  retrieval. 

a.  The  thesaurus  should  not  only  be  examined  but 
also  tested  by  competent  General  Counsel  attorneys 
for  its  potential  retrieval  efficiency. 

b.  This  editing,  reviewing,  and  testing  should,  In 
turn,  general  suitable  generic  relationships  and 
cross-references. 

c.  A  revised  thesaurus,  incorporating  the  aforemen¬ 
tioned  term  "structures"  should  be  developed  at 
a  good  first  cut  of  a  firm  efficient  vocabulary 
control  over  future  indexing. 

d.  It  muct  be  remembered  that  this  latter  thesaurus 
will  le  only  a  "good  first  cut."  It  should  always 


A  2 


remain  open-ended  (subject  to  change  and  modi¬ 
fication,  but  only  under  rigidly  controlled 
conditions). 

e.  Further,  this  thesaurus,  analyzed  in  conjunction 
with  the  indexed  documentation,  will  teach  many 
lessons  about  the  approach  and  depth  of  Indexing 
required . 

The  legal  opinions  and  legal  memoranda  Indexed  ori¬ 
ginally  in  the  sample  should  be  re-indexed  to  correct 
deficiencies  indicated  in  the  procedure  above. 

The  revised  indexing  terminology  from  the  sample 
should  oc  converted  into  information  retrieval  hard¬ 
ware  . 


Depending  upon  the  success  and  economic  trade-offa 
of  the  above  effort,  we  conditionally  recommend  that 
Congressional  materials  be  considered  for  indexing 
by  their  "administrative  handles,"  such  as  Titles, 
Types  of  publication.  Publication  numbers.  An 
"administration  legal  thesaurus"  would  then  be  gen¬ 
erated  and  follow  through  a  similar  set  of  steps 
indicated  in  2  above. 

Criteria  should  be  established  so  that  each  function 
and  component  of  the  experiment  can  be  evaluated. 

These  criteria  are  a  prerequisite  to  realistic  and 
intelligent  trade-offs.  Appropriate  decisions  (and 
molementat ion  of  these)  can  then  be  made  on  over- 
.1  system  approach,  hardware,  procedures  and  timing; 
and  on  back  "ogs  cf  legal  opinions,  legal  memoranda 
and  Congress  nal  materials. 

The  experiment  should  explore  coordination  problems 
with  Regional  Offices  (and  their  part  in  the  program), 
updating  procedures  and  other  areas  not  immediately 
obvious  and  critical  here. 
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Figure  B-3 
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Figure  B-4 
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5)  Includes; 

1 

includes : 

6)  Speoifio  to; 

SCIENTIFIC  RESEARCH 

■ 

Speoifio  to : 

i 

7)  Generic  to: 

■ 

Generic  to : 

i 

8)  ALSO  SEE: 

CRYOGENICS 

COLD  WEATHER  TESTS 

1 

ALSO  SEE: 

Indexer's  ooament:  The  term  ia  included  in  the  ASTI A  Supplement,  and  is 
needed  for  indexing  where  CRYOGENICS  (already  a  descriptor)  does  not 
apply.  As  a  precedent,  the  Thesaurus  already  has  HIGH-TEMPERATURE 
RESEARCH. 


Disposition  by  Review  Committee: 

Approved,  March  24,  1965 
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Figure  B-5  -  SAMPLE  CATALOG  CARD 
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Figure  B-6  -  SAMPLE  IDENTIFIER  CARD 


Figure  B-7 


PROCEDURE  FOR  PREPARING  INDEX  FILES 
FOR  INSTRUCTION  BOOKS  AND  MANUALS 


*In  this  step  of  the  procedure,  duplicate  manuals  were  screened  and  location  codes 
were  added  to  the  master  card.?. 
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FA-WA-4534  Generator*.  Thr««  Witt*  Engine  Work* 
Fh*M  Oil  Hall  Supply 

D It. 

Witte  Mini  Engine-Canerator  Dolt*.  Instruction 
Manuel.  Juna  18,  19Cj,  FA -VA -45 54. 
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Figure  B-8 

SAMPLE  INDEX  CARD  FOR  INSTRUCTION  MANUALS 
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Figure  B-9  -  SAMPLE  SEARCH  QUE8TI0N  RECORD 
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Figure  B-10 

REVISED  SEARCH  QUESTION  RECORD 
Search  No.  isearcher.  I  Requestor  (Name.  Address,  Telephone  No.): 


Date  and  Time 

Deadline: 

Number  of 

Date  and  Time 

of  Request: 

References 

Desired: 

Completed: 

Complete  Search  Question  (Narrative  form,  specifying  what  is  to  be  included 
ZL  or  excluded): 


References  Retrieved  (Continue  on  reverse  side  if  necessary): 

D 
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Contractual  Obligations 

This  paper  sots  forth  the  details  of  the  test  methodology  that  JONKER 
Business  Machines,  Inc.  will  undertake  to  comply  with  the  obligations  of 
Contract  FA  64-WA-5186:  Investigation  into  the  Retrieval  Indexing  and 
Searching  System.  An  excerpt  of  that  contract's  specifications  appears  below.* 

"A  testing  program  shall  be  developed  and  carried  out  to  test 
the  effectiveness  of  the  document  retrieval  system  (Termatrex)  in 
terms  of  its  recall  power  (i. e . ,  what  proportion  of  pertinent  documents 
in  the  system  are  retrieved  in  response  to  a  request?)  and  relevance 
power  (how  much  "noise"?).  The  test  of  the  effectiveness  of  the  FAA 
system,  in  terms  of  recall  and  relevance,  shall  provide  information  on: 

The  adequacy  of  the  present  depth  and  specificity  of  index¬ 
ing,  any  weakness  of  the  present  Thesaurus  as  a  searching  tool, 
the  efficacy  of  generic  posting,  and  optimum  search  procedure. 

"The  testing  program  developed  shall  be  subject  to  final  review 
and  approval  by  the  Project  Officer." 

The  FAA  Information  System 

The  Federal  Aviation  Information  Retrieval  System  (FAIRS)  is  located 
in  Washington,  D.  C. ,  where  it  serves  users  who  have  a  diversity  of  needs 
for  technical  information.  Through  a  cooperative  arrangement  embotjying 
centralized  processing/decentralized  operation,  the  Agency  has  made  its  in¬ 
formation  available  to  other  users  at  its  remote  Regional  Offices. 

When  the  FAIRS  is  tested  for  retrieval  effectiveness,  it  will  have  been 
in  operation  for  about  a  year  and  will  comprise  a  collection  of  10,  000  technical 
reports  covering  the  gamut  of  science  and  technology  related  to  aviation. 

These  documents,  numbered  and  filed  serially,  have  been  prepared  for  re¬ 
trieval  by  coordinate  indexing  from  a  controlled  thesaurus  of  nearly  3,  200 
descriptors;*  an  average  of  9  descriptors  have  been  assigned  to  each  report. 

The  major  retrieval  device  for  the  FAIRS  is  a  manually  operated, 
browseable  Termatrex  system.  A  search  question  is  answered  with  Termatrex 
by  superimposing  the  applicable  cards  (one  for  each  descriptor)  over  a  light 


I  ‘From  page  2  of  the  Negotiated  Contract,  FA  64-WA-5186. 
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source,  then  reading  the  report  numbers  that  are  yielded. 

Some  other  features  of  the  FAIRS  have  special  bearing  on  the  test 
environment, 

1.  At  the  present  time,  a  large  indexing  project  is  underway.  This 
on-going  project  is,  in  it3elf,  a  test-bed  for  hypotheses  and 
theories  that  come  from  the  experimental  tests.  For  example, 
indexing  can  be  redirected  to  levels  of  more  (ot  less)  exhaustivity 
or  more  (or  less)  specificity. 

2.  The  Thesaurus  of  FAA  Descriptors,  used  for  indexing  and 
searching,  is  continually  being  updated  and  developed.  Changing 
its  structures,  expanding  its  scope,  or  altering  its  format  remain 
feasible. 

3.  Within  the  Thesaurus,  moreover,  the  relationship  among  generic- 
specific  descriptors  is  used  for  a  mandatory  "generic  posting" 
during  Termatrex  card  drilling.  That  is,  the  card  for  a  generic 
term  is  drilled  when  any  of  that  term's  specific  terms  have  been 
selected  for  indexing.  This  feature  of  the  system's  input  is  to  be 
studied  during  the  tests  so  that  its  merit  can  be  decided. 

4.  Indexing  and  searching  are  conducted  within  proximity  of  each 
other  in  the  FAA  Library.  The  convenient  interface  between 
these  two  functions  facilitates  the  investigation  of  search  strategies 
and  user  acceptance,  among  others. 

5.  A  reference  file  has  been  maintained  during  the  past  several  years 
to  contain  the  actual  search  questions  that  have  been  asked  of  the 
system.  Those  records  will  be  useful  in  providing  the  names  of 
FAA  employees  who  might  participate  in  the  tests.  Examination 
of  that  file  will  also  reveal  how  frequently  the  system  is  used  and 
what  subject  matter  its  report  collection  is  expected  to  provide. 
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Experimental  Testing 

In  recent  years,  the  number  of  systems  and  devices  for  storing  and 
retrieving  technical  information  has  been  growing  apace  with  the  expanding 
volume  of  published  information- -a  common  biproduct  of  our  proliferating 
research  and  development  activity.  But  in  a  society  so  conscious  of  its  rank 
in  the  world's  technology,  demands  for  more  efficiency  are  being  levied 
against  the  people  who  manage  that  technical  literature. 

While  in  some  instances  these  demands  have  been  belatedly  imposed, 
in  almost  all  instances  they  have  been  difficult  to  satisfy.  For  one  reason, 
the  Laws  of  Vested  Interests  prevent  "inferior"  systems  from  being  exposed. 
Another  reason  lies  with  confusing  intricacies  of  testing  any  information 
system — a  confusion  that  even  the  experts  have  not  settled.1* 

From  the  very  beginning  of  the  era-of-search-for-the-efficient- 
system,  in  1953,  M.  Taube  hoped  that  "further  research  by  ourselves  and 
others  will  lead  not  only  to  tests  of  consumer  satisfaction  with  various 
systems,  but  also  to  a  more  systematic  presentation  of  the  interval  criteria 
of  evaluation."2 

Taube's  hopes,  however,  were  not  fulfilled.  "Further  research" 
continued  on  a  host  of  system  attributes,  usually  expressed  as  some  form 
of  cost,  speeu,  volume,  and  reliability,  but  the  tests  were  often  conducted 
without  controlled  variables  and  without  standardized  techniques. 

Ten  years  later,  R.  R.  Shaw  assessed  the  state  of  affairs  in  bitter 
words.  "It  has  yet  to  be  demonstrated  that  those  who  have  been  crying  havoc 
and  calling  for  vast  expenditures  have  anything  to  offer  that  will  currently  In¬ 
crease  tho  effectiveness  of  our  information  retrieval  services."'* 

On  the  matter  of  testing,  Shaw  called  for  an  end  to  "claims  for  what¬ 
ever  system  is  being  advocated,  on  the  basis  of  facts  not  given  and  by  com¬ 
parison  with  the  worst  alternatives." 

Aware  of  the  "articulate  proponents  for  different  systems  and  of  the 
general  interest  in  IR  systems"  in  1959,  the  National  Science  Foundation 
sponsored  a  study  that  produced  the  first  accumulation  of  data  from  an  ex¬ 
perimental  test  involving  what  they  called  a  genuine  scientific  method.® 
Theirs  was  a  comparative  study  of  the  efficiency  of  four  different  systems. 
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As  partial  financial  support,  the  Foundation  granted  $159,  200  to  Western 
Reserve  University. 


Another  grant  of  $16,700  went  to  the  Association  of  Special  Libraries 
and  Information  Bureaus  (AS LIB)  for  what  has  become  known  as  the  Cranfield 
Research  Project.  This  latter  grant  provided  for  a  separate  analysis  of  two 
of  the  four  systems,  which  was  also  published  separately  in  one  of  the  most 
detailed  reports  yet  to  be  written  on  the  subject.6 

Relevance  and  Recall 


In  the  Cranfield  study,  two  systems  were  subjected  to  the  same 
questions,  with  each  searching  among  documents  that  were  common  to  their 
collections.  The  efficiency  of  each  was  measured  according  to  how  well  It 
retrieved  relevant  documents  and  how  well  it  avoided  non -relevant  documents. 
The  criteria  for  comparison,  then,  were  two  variables  names  Relevance  Ratio 
and  Recall  Ratio. 

The  basic  ingredients  for  these  two  ratios  are  illustrated  in  the  2  x  2 
contingency  table  below;  for  consistency,  the  symbols  expressed  in  the  table 
are  the  same  as  those  used  for  the  Cranfield  study. 


Figure  C-l  -  RELEVANCE  AND  RECALL  CONTINGENCY  TABLE 
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From  the  table,  relevance  ratio  can  be  expressed  as  follows: 

D 

Relevance  ratio  =  100  x  - 

L 


Where  R  is  the  number  of  relevant  documents  retrieved  for  a  search  question, 
while  L  is  the  total  number  of  documents  retrieved  for  that  question. 

Recall  ratio  can  be  expressed  as: 

Recall  ratio  -  100  x  - 
C 


Where  R,  again,  is  the  number  of  relevant  documents  retrieved  for  a  search 
question  and  C  is  the  total  number  of  documents  in  the  collection  that  have  an 
agreed  standard  of  relevance . 

Although  Cranfield's  Relevance  and  Recall  Ratio's  were  only  2  of  at 
least  10  other  measures  developed  during  the  era, 7  they  were  given  an 
attentive  reception  of  both  praise  and  criticism  by  Documental ists.  One 
caustic  critic  predicted  an  early  death  for  these  test  criteria  and  speculated 
that  their  use  would  cause  "many  current  studies  to  be  looked  upon — in  the 
course  of  history— as  comparable  to  the  epicycles  of  the  14th  century."8 

Ironically,  the  zealous  search  for  a  scientific  method  for  testing  the 
performance  of  IR  systems  led  to  the  criticisms,  for  it  has  been  the  non- 
scientific  aspects  of  those  studies  which  have  drawn  the  critics^  fire.  Their 
remarks  have  been  lured  primarily  against  those  proponents  who  attribute 
too  much  validity  to  the  criteria  of  relevance  and  recall,  especially  investi¬ 
gators  who  fail  to  recognize  the  subjective  and  non-mathematical  nature  of 
the  two  entities . 

At  the  heart  of  the  controversy  has  been  what  Cranfield  spoke  of  as 
an  "agreed  standard  of  relevance."  Relevance  was  determined  by  subject 
specialists  who  reviewed  not  only  every  document  that  the  system  retrieved, 
but  also  those  the  system  did  not  retrieve  (over  100,000  individual  assess¬ 
ments'.)  .  These  Judges  used  a  simple  standard  to  gauge  relevance:  was  the 
report  as  good  as  the  "source"  document  from  which  the  search  question 
was  originally  inspired? 

The  flaw  in  judging  relevance,  say  the  critics,  is  that  judges  cannot 
possibly  determine  a  report's  relevance  to  an  information  need,  which  is  to 
say  that  "non-users"  cannot  allow  for  uie  difference  between  an  asserted 


need  (a  written  statement  of  request)  and  a  real  need.  Real  needs  can  be 
measured  only  indirectly,  by  the  user  they  say,  and  then  with  great  uncer¬ 
tainty. 

The  truth  of  the  matter  is  that  no  one  knows  the  correlation  between 
a  request  and  a  need.  Needs  are  seldom  clear  and  are  rarely  stable;  the 
same  may  surely  be  said  about  requests.  They  are  all  victims  of  time  and 
circumstances.  It  should  be  recognized,  then,  that  a  successful  information 
system  must  allow  for  a  user's  change -of-heart.  (This  argument,  some 
say,  is  a  strong  one  for  browseable  retrieval  systems,  such  as  Termatrex.) 

A  Decision 

Despite  the  controversy  about  testing  information  retrieval  systems 
(which  could  last  another  decade),  a  decision  has  been  made  to  use  the  sub¬ 
jective  Relevance  and  Recall  Ratios  in  the  present  tests.  That  choice  is 
not  merely  to  satisfy  the  contractual  obligations  (which  call  for  a  testing 
program  in  terms  of  "Recall  power"  and  "Relevance  Power  "),  but  because 
relevance  and  recall  are  believed  useful  in  assessing  the  internal  workings 
of  an  information  system .  A  proper  realm  for  relevance  and  recall  in  the 
FAIRS  is  in  drawing  attention  to  the  intellectual  aspects  of  indexing,  the 
adequacy  of  a  thesaurus,  and  the  strategies  for  searching.  They  should  pro¬ 
vide  information  for  those  who  ar®  trying  ti  re-engineer  and  improve  the 
operation  of  the  system. 
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The  Proposed  Test 

This  se  ction  of  the  paper  presents  the  plan  for  conducting  an  experi¬ 
mental  test  of  the  Federal  Aviation  Information  Retrieval  System.  In  brief, 
the  test  will  evaluate  how  well  the  system  retrieves  reports  that  are  relevant 
to  search  questions.  The  procedure  of  the  test  is  to  search  the  entire  10,000- 
document  collection  with  questions  that  will  be  randomly  selected  from  those 
previously  submitted  by  FAA  personnel  for  actual  searches. 

It  is  being  proposed  that  these  same  subject  specialists  judge  die  re¬ 
ports  that  are  retrieved  for  relevance  to  their  particular  search  question. 
When  the  reports  have  been  assessed,  the  performance  of  the  system  will  be 
stated  in  terms  of  its  Relevance  Ratio  and  Recall  Ratio,  while  further  investi¬ 
gations  will  seek  the  deficiencies  of  the  system  for  1)  retrieving  non-relcvant 
documents  and  2)  not  retrieving  all  the  relevant  documents.  As  part  of  the 
stucfy,  recommendations  for  eliminating  these  system  deficiencies  will  be 
made. 


The  test  procedure,  illustrated  on  the  next  page,  is  discussed  in 
greater  detail  in  the  paragraphs  that  follow. 

Documents  for  the  Study 

The  entire  FAIRS  collection  of  U>,  000  technical  rei-orts,  which,  for 
convenience  will  be  referred  to  as  Document  Set  A,  will  bt  searched  during 
the  tests.  It  is  estimated  that  Document  Set  A  might  yield  CO  reports—on  the 
average--for  eacn  question  imposed  against  it.  Every  one  of  those  reports 
will  be  judged  for  its  relevance  to  the  respective  questions.  (These  assess¬ 
ments,  s.ocdc i  ic  Je:  iw  -»c  n,*oi.em  h  Relevance  Ratio,  should  require  about 
one  hour  per  quest  ion .  t 

Although  the  proposed  test  procedure  calls  for  an  ass  ssment  of 
every  document  that  is  retrieved  by  searching,  it  does  not  require  that  the 
entire  report  collection  be  judged  to  identify  relevant  reports  that  were 
missed  in  searching.  The  quantity  of  relevant  reports  that  are  missed  by 
the  system  must  be  measured  in  order  to  derive  the  Recall  Ratio,  but  to 
judge  every  document  in  the  collection  against  each  question  is  hardly  a 
practical  method.  To  assess  each  of  the  lo,  000  documents  could  easily  ab¬ 
sorb  200  man-hours  per  question!  A  tactful  limit  on  that  burde.i  to  profes¬ 
sional  people  should  uc  no  more  than  S  rr.an-hmirs  per  question.  What  is 


figure  C-2 

PROCEDURE  FOR  EFFECTIVENESS  STUDY 


Recommend  system 
improvements 

Recomtiend  a  re. is 
for  further  study 

rr'z-'zrvTr— 


j  Write  final  report 
of  stud-. 


Approve  final 
report 


r.:t 


i 


equally  distressing  under  such  a  procedure  is  that  most  of  that  effort  would 
be  non -productive.  A  few  words  of  explanation  here  will  help. 

We  know  that  the  information  system  at  FAA  is  specifically  designed 
to  satisfy  the  needs  in  aviation,  and  the  existence  of  the  FAIRS  is  justified — 
in  part— by  the  lack  of  another  information  system  in  the  Federal  Government 
devoted  to  that  specialty:  others  offer  indexing  and  vocabularies  that  are  too 
general  to  fit  the  Agency's  needs.  The  Defense  Documentation  Center,  for 
example,  currently  categorizes  its  reports  into  22  fields  of  which  one  is 
"Aviation;"  NASA  uses  34  categories,  Including  one  for  "Aircraft."  The  rest 
of  their  collections  are  allied  to  many  fields  that  are  neither  aviation  nor 
aircraft. 

We  might  quickly  conclude  from  these  comparisons  that  the  information 
in  the  FAIRS  is  specialized,  whereas  information  in  the  others  is  not.  Against 
one  frame  of  reference,  that  supposition  is  valid;  yet  within  "aviation"  (to 
change  the  frame  of  reference),  there  exists  a  great  diversity  of  subjects,  such 
as  Air  Traffic  Control,  Display  Systems,  Weather  Forecasting,  Navigational 
Aids,  Airport  Management,  Supersonic  Transports,  Hazards  and  Safety,  Pilot 
Training,  and  Aerodynamics.  With  a  little  effort,  the  list  of  subjects  strongly 
related  to  aviation  could  be  lengthened  to  show  clearly  that  the  FAA  collection 
must  satisfy  a  variety  of  specialized  needs  within  a  specialized  field.  The 
FAA  collection  of  reports  can  be  considered  specialized  only  if  viewed  from 
without,  while  general  if  viewed  from  within. 

That  anomoly  has  particular  bearing  at  the  moment,  because  it  has 
prompted  us  to  make  a  "ballpark  estimate"  about  the  number  of  reports  (50) 
that  are  likely  to  be  retrieved  for  any  one  question.  From  our  experience 
with  the  system,  we  are  confident  in  speculating  that  for  any  single  informa¬ 
tion  need,  for  a  question  imposed  upon  the  system,  at  least  75%  of  the  reports 
in  the  collection  are  not  applicable.  Reversing  the  math<  -matics  makes  more 
sense:  for  any  given  question,  the  collection  could  not  reaiw....  yield  2,500 
relevant  reports. 

During  the  conduct  of  the  test,  as  in  "real’’  searching,  it  is  important 
to  avoid  burdening  professional  men  with  the  entire  shelf  of  reports  when  we 
know  well  ahead  of  time  that  only  a  handful  of  th-;m  could  possibly  be  relevant 
to  his  need.  By  practical  necessity,  the  proposed  plan  intends  to  reduce  that 
imposition  and  to  do  it  simply  by  reducing  the  number  of  (non- retrieved) 
reports  that  must  be  judged. 

One  early  consideration  was  to  assess  all  the  reports  that  occurred 
in  a  set  of  only  100  randomly  chosen  documents,  but  that  sample  was  too 
small;  it  would  not  contain  enough  reports  relevant  to  any  question,  thus 
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failing  to  provide  sufficient  statistical  data .  To  collect  ample  data  for  the 
system's  Recall  Ratio,  a  10%  sample  of  1, 000  documents  is  more  appropriate, 
but  as  much  as  20  hours  per  question  could  be  required  to  assess  them — an 
effort  again  too  burdensome .  To  reach  a  satisfactory  workload  without 
destroying  the  data  base  for  the  tests,  it  is  being  proposed  that  the  10%  sample 
still  be  used,  but  that  only  a  portion  of  its  reports  be  judged  for  each  question: 
the  1,000-document  sample  (Document  Set  B)*  will  be  "screened"  to  eliminate 
irrelevant  reports. 

Screening  does  not  involve  direct  human  judgment  of  documents  as  it 
usually  implies,  but  will  be  done  by  selecting  descriptors  from  the  Thesaurus 
for  each  question.  The  applicable  descriptors  will,  in  combination,  define 
the  scope  of  a  given  question's  subject.  (As  a  rule  of  thumb,  when  there  is 
doubt  about  the  applicability  of  a  term,  that  term  will  be  added  to  the  list.) 

The  reports  that  will  be  submitted  for  relevance  apses  snts  for  the  given 
question  will  be  those  reports  that  have  any  one  of  the  chosen  descriptors. 
Reports  without  even  one  will  be  assumed  to  be  irrelevant.  An  example  will 
help  illustrate  this  screening  technique , 

Suppose  die  question -at-hand  sought  information  ou  the  "public  reac¬ 
tion  to  sonic  boom  and  aircraft  noise."  A  cursory  review  of  the  Thesaurus 
reveals  at  least  11  descriptors  that  might  te  related  to  the  subject  of  that 
question.  The  descriptors  and  their  respective  posting  densities  are  listed 
in  Figure  C-3. 

The  total  posting  density  of  1060  should  be  doubled  to  accomodate  die 
effects  of  generic  postings;  and  of  tuat  new  total,  about  212  postings  (10%  of 
2,120)  would  likely  appear  within  the  randomly  chosen  technical  reports  of 
Document  Set  B. 


Figure  C-3  -  SAMPLE  POSTING  DENSITIES 


Descriptor 

Posting 

Density 

AIRPLANE  ENGINE  NOISE 

16 

AIRPLANE  NOISE 

64 

SONIC  BOOM 

69 

ENGINE  NOISE 

7 

NOISE 

276 

JET  PLANE  NOISE 

63 

EDUCTION 

I  20 

SUPPRESSORS  (ACOUSTIC) 

40 

JET  PLANES 

169 

JET  TRANSPORT  PLANES 

165 

JET  ENGINES 

51 

TOTAL 

10€0 
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So  if  this  were  an  actual  search  question  and  if  the  list  of  applicable 
i  descriptors  were  complete,  the  212  reports  from  Document  Set  B  would  be 

submitted  to  the  FAA  staff  member  for  his  relevance  assessments,  along 
y.'lth  the  50  reports  that  would  be  retrieved  from  Document  Set  A. 

Questions  for  the  Study 

Ten  search  questions  will  be  randomly  selected  from  the  actual,  past 
questions  that  have  been  recorded  on  the  Search  Question  Records  (FAA 
Form  2712)  and  which  are  retained  in  the  files  of  the  Information  Retrieval 
Branch.  As  the  first  step  in  selecting  test  questions,  die  Records  have  been 
reviewed;  the  subjects  of  192  candidate  questions  are  listed  in  Figure  04 
at  the  end  of  this  Appendix.  For  a  question  from  that  list  to  be  approved  for 
the  test,  its  subject  must  have  remained  of  current  value  and  interest  to  the 
respective  user.  A  question  would  be  discarded,  too,  if  its  "author"  were 
unwilling  to  participate  in  the  tests . 

Searching 

The  key  concepts  in  each  of  the  10  test  questions  that  are  selected 
will  be  identified  by  a  list  of  descriptors,  from  those  used  for  "screening" 
within  Document  Set  B.  For  each  question,  four  different  search  strategies 
will  then  be  developed  and  consistently  used,  regardless  of  how  *he  searches 
were  previously  made  by  the  FAA  Library  staff.  These  strategies  will  be 
designed  to  control  the  effects  of  generic  posting,  thus  permitting  the  ade¬ 
quacy  of  the  Thesaurus  and  the  indexing  techniques  to  be  studied. 

Searching  during  the  tests  will  be  done  onanua’ly  on  ;he  Termatrex 
system,  and  the  accession  numbers  of  reports  that  are  retrieved  for  each 
question  will  be  recorded  so  that  the  hard-copy  reports  can  be  pulled  from 
the  Library  shelves  and  loaned  to  the  FAA  participant. 

Relevance  Judgments 

The  entire  text  of  each  report  will  be  rev'ewed  fc"  itt  relevance  or 
non-relevance  to  the  respective  search  question.  The  standard  for  gauging 
relevance  should  be: 

"Does  the  report  provide  information  which  has  direct 

bearing  on  the  search  question?" 

Notations  of  relevance  (yes)  or  non-relevance  (no)  will  be  made  by 
the  judges  onto  the  accession  number  lists  that  will  have  been  developed 
during  searching. 
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Relevance  assessments  for  the  total  of  about  2, 600  technical  reports 
should  require  about  52  man-hours,  or  about  5  hours  per  question.  That 
estimate  is  based  on: 

500  technical  reports  from  Document  Set  A,  at  50  reports  retrieved 
for  each  of  10  search  questions. 

2,100  technical  reports  from  Document  Set  B,  at  210  reports  screened 
for  each  of  10  questions . 

Relevance  and  Recall  Ratios 

Relevance  and  Recall  Ratios  will  be  calculated  for  each  test  question, 
and  then  for  the  system  by  cumulating  the  individual  results.  These  over-all 
ratios  provide  a  basis  for  comparing  the  FAIRS  to  other  systems,  to  the 
same  ratios  developed  during  other  experimental  tests. 

The  tabulations  used  throughout  the  tests  will  have  facilitated  these 
calculations,  because  they  will  have  denoted: 

Relevant  documents  retrieved 
Relevant  documents  not  retrieved 
Non-flelevant  documents  retrieved 

Reasons  for  Failure 

Since  these  tabulations  have  identified  the  system's  success  and 
failure  about  specific  technical  reports,  it  shall  be  possible  to  determine 
the  causes  of  retrieval  failure  by  examining  the  documents  and  their 
Document  Analysis  Worksheets.  Document  inspection  is  a  key  step  in  ex¬ 
ploring  the  internal  workings  of  the  system,  and  it  shooM  do  much  to  measure 
the  effects  of  generic  posting,  the  adequacy  of  the  Thesaurus,  :  strategy 
for  searching,  and  the  techniques  of  indexing.  To  reflect  on  these  major 
ijreas  of  the  system's  operation,  the  "causes  of  failure"  have  been  so  organ¬ 
ized  in  tentative  checklists:* 


*  These  checklists  are  not  final.  In  fact,  they  will  be  generated  after  the 
documents  have  been  examined-  not  before— so  that  an  analytical  view  can 
prevail . 
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RELEVANT  BUT  NOT  RETRIEVED 


INDEXING 

Concepts  omitted 
Concepts  Indexed  too  generically 
Concepts  indexed  too  specifically 
Clerical  errors  in  coding 

SEARCHING 

Specificity 

Exhaustivity 

THESAURUS 

More  then  one  descriptor  for  the  concept 
Inadequate  references  to  comparable  concepts 
No  descriptor  for  the  concept 
Restriction  imposed  by  Descriptor  Group 
Generic  posting 


NGN-RELEVANT  BUT  RETRIEVED 


INDEXING 

Too  exhaustive 
Too  specific 


SEARCHING 


Concepts  sear  died  but  only  implied  by  question 
Concept  too  generic 
Concept  omitted  from  search  strategy 
Improper  coordination  of  search  terms 

THESAURUS 

Concept  too  generic  (from  generic  posting) 
Term  too  abstract 
Specific  term  lacking 
Intermediate  term  lacking 


68 


Documentation 


The  results  of  the  study  will  be  documented  in  a  special  section  of 
the  final  report  and  will  i>c  in  a  format  suitable  to  compare  the  topics 
covered  by  the  Cranficld  Research  Project®  and  by  Atherton's  reporting 
standards. 
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Figure  C-4 

SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

020 

021 

023 

024 

026 

029 

032 

034 

038 

218 

035 

037 

008 

012 

031 

085 

Oil 

189 

001 

002 

t 

003 
005 
196 
220 


Brief  Description  of  Subject 
Aircraft  design  for  airconditioning 
NAFEC  Fire-control  reports 
Aerial  photography 
Helicopter  pilots 
Bright  displays  for  ATC 
Tire -hydroplaning 
Snow  removal 

FAA  policy,  per  Halaby  speeches 
Control  of  birds  by  sound 
National  Runway  markers 
Delays  at  National  Airport 
Paper  by  Botte 

Accuracy  of  air  traffic  controllers 

Radiofrequency  mission  of  thunderstorms 

Efficiency  of  aircraft  production  industries 

Reliability  of  Semiconductors 

FAA;  Invetn-  'v 

FAA  installations 

Propeller  blades 

Icing,  snow  on  VOR 

Reliability  of  electronic  components 

Stall 

V0R/DME 

Stall 
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SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

Brief  Description  of  Subject 

093 

Hazard  from  rocket  launches 

094 

Speeches  by  Halaby 

095 

Coat  and  accidents 

096 

Air  cargo  security 

100 

Pilot  Proficiency 

101 

Noise  prediction 

103 

Alpha  numeric  displays 

104 

Runway  roughness 

105 

ATC 

106 

ATC  age 

107 

ATC  simulators 

108 

Swept -wings 

109 

Eirdc  and  collision 

110 

Hydrofoils 

111 

SST 

112 

Weather  and  sonic  boom 

113 

ATC 

250 

Cost  effectiveness;  personne 

251 

Crash  recorder 

252 

Voice  recorder 

253 

Crash  locator  Beacon 

254 

Columbian 

267 

Concorde  airplane 

255 

Simulators;  FAA 
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SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

188 

050 

052 

053 

054 

055 

056 

058 

059 

060 

061 

070 

071 


Brief  Description  of  Subject 
FAA 

Aircraft  noise 
System  Reliability 
Batteries 

Student  exam  failures 

Near  misses 

Noise  alternation 

Statistics  re  airports 

Transponders 

Contract  officers 

ILS  use  by  pilots 

Economics  of  Air  Transportation 

SST 


072 

073 

074 

0H5 

086 

087 

088 

089 

090 

091 

092 


One  Report 
Tying  aircraft  down 
Aircraft  and  Sand  Strip 
Reliability  of  semiconductors 
ATC  centers 

Shock  vaves  from  supersonic  aircraft 
Reliability  of  electronic  components 
Sonic  boom 

An  airplane's  airconditioning  and  vindov  design 

BLEU  I 

Fog 
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SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Quest  ion 
Number 

Brief  Description  of  Subject 

150 

Precipitation 

151 

Airborne  radios 

153 

Anti-skid  devices 

154 

Runway  roughness 

155 

Anti -locking  device 

160 

Use  of  computers 

161 

SST  from  1950's 

162 

Sonic  boom 

163 

Flight  plans 

164 

Safety  speeches  by  Halaby 

165 

Duration  of  vlndo 

166 

History  of  ATC  in  the  1950's 

167 

Subsonic  jets  In  the  1950's 

170 

ATC 

172 

ATC 

173 

Positive  control 

174 

Density  altitude 

175 

Collision  avoidance 

176 

SST 

177 

Aircraft  fuel 

178 

Instrument  landing  approach 

179 

Ocean  ATC 

180 

Nose-loading  aircraft 

186 

Aging  of  man 
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SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

Brief  Description  of  Subject 

230 

Antenna  side-lobe  suppression 

231 

Aircraft  lighting  and  marking 

232 

Decompression  in  aircraft 

234 

ATC  data  handling  and  display 

235 

Maneuvering  in  final  approach 

236 

Airport  management 

120 

Blue  lights  in  radar  rooms 

121 

Statistical  analysis 

122 

Man-machine  systems 

222 

Snov,  ice  hazards 

223 

Bird  hazards 

125 

Ocean  ATC 

126 

Bright  lights 

127 

CO  detection 

224 

Airborne  antennas 

130 

Fuel  tanks 

133 

Frequency  spectra  of  gusts 

135 

SST  Navigation 

209 

CCA 

140 

Airport  delays 

141 

Doppler  radar 

142 

Radar  remoting 

143 

Blind  spots  at  airports 

225 

ATC 
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SUBJECT  ANALYSIS  OF  SEARCH  QUESTION 


Question 

Number 

Brlif  Description  of  Sublect 

256 

Airworthiness 

257 

Aircrago 

258 

Radar  Beacons 

259 

Radar  Reflectivity 

260 

TACAN 

261 

Engine  Failure 

262 

Speech  by  Halaby 

263 

Microminiaturization 

264 

Radio  isotopes 

265 

Alpha  numerics 

266 

Pricing;  aimorts 

270 

Aircraft  noise 

271 

Turbulence  and  Helicopters 

272 

V/STOL  lift  fan 

273 

Slot  Antennas 

274 

Halaby 

275 

Dean 

276 

Birds  and  runways 

277 

VOR/DKE 

279 

Digital  Cotmsun lea t ions  Testing 

280 

Aptitude  tests 

281 

Matheautical  models;  radar 

282 

Cost  Savings ;  ATC 

278 

Approach  landings 

019 

Aircraft  Navigation  Systems 
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SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

233 

229 

200 

195 

197 

198 

199 
006 
007 
013 
014 
015 
016 
017 
018 

300 

301 

302 

303 

304 

305 

306 

307 


Brief  Description  of  Subject 
Arresting  for  aircraft 
Constant  Mach  numbers  for  cruising 
Population  trend,  vehicle  trends 
Aircraft  tires 
Statistical  detection,  digital  coramunicati jn 
All  FAA  reports  on  weather 
Reports  on  propeller  blades 
ATC  training,  accuracy 
Runway  arresting  devices 
N.J.  and  N.Y.  iirnorts  and  jetports 
Aircraft  navigation,  VOR  and  VORTAC 
B-707  VORTAC 
Daylight  lights 
FAA;  decision-making 
Aircraft  position  lights 
Noise  in  ATC  towers 
Cost  of  All-weather  Landing 
Long-Range  Planning 
Airport  Management 
Materials  to  Remove  Runway  Glare 
Time  Effectiveness 
B-720  Response  to  Runway  Lighting 
Integrated  Circuits 


SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

309 

310 

311 

312 

313 

314 

316 

317 

318 

319 

320 

321 

322 

323 

324 

325 

326 

327 

328 

329 

330 

331 

332 


Brief  Description  of  Subject 
Microwave  Propagation 
Defrultlng 

TACAN  (an  Identifier) 

Project  Searchlight 

Air  Pollution 

Short-haul  Aircraft 

Effects  of  Oxygen  on  A/C  Crews 

Terminal  Area  Air  Traffic  Operations 

Fog  Dissipation 

Radar  Resolution 

Reliability;  Maintainability;  Quality  Control 
VORTAC 

Aviation  in  Emerging  Nations 

A/C  and  Air  Pollution 

Steep-Gradient  Aircraft 

Decentralization 

Cost  Effectiveness 

Reliability  of  Transistors 

Air  Traffic  Control  Bibliography 

Air  Traffic  Controllers  (Retirement) 

Air  Traffic  Controllers  (Stress) 

Statistical  Analysis 
PERT 


SUBJECT  ANALYSIS  OF  SEARCH  QUESTIONS 


Question 

Number 

333 

334 


Brief  Description  of  Subject 
Cosanunication  and  Navigation  Alda 
Capture  Effect  Glide  Scopes 
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GLOSSARY 


(Definitions  that  fit  the  Federal  Aviation  Information  Retrieval  System) 


CONCEPT  INDEXING:  The  intslleotual  process  of  choosing  the  concepts  in 
a  particular  document  or  searoh  question  that  are  of  sufficient  im¬ 
portance  for  retrieval. 

EXHAUSTIVTTY i  The  selection  of  many  ooncepts  during  concept  Indexing. 

POST  ALSO:  An  order,  dictated  by  the  thesaurus,  which  requires  that 
generic  terms  be  posted  in  the  file  (Terms trex  cards)  when  any  of 
their  respective  specific  terms  had  been  selected  for  indexing. 

RECALL:  fcben  the  word  is  used  alone,  it  becomes  synonymous  with 
retrieval. 

RECALL  RATIO:  The  number  of  relevant  documents  retrieved  for  a  search, 
divided  by  the  total  number  of  relevant  documents  in  the  collection; 
multiplying  by  100  gives  recall  ratio  in  a  percentage. 

RELEVANCE:  A  qualitative  trait  of  a  document  having  "direct  bearing" 
on  a  particular  searoh  question. 

RELEVANCE  RATIO:  The  number  of  relevant  documents  retrieved  for  a 
searoh,  divided  by  the  total  number  of  documents  retrieved  for  that 
searoh. 

SPECIFICITY:  Concept  indexing  that  choo3es  terms  that  are  co- extensive 
with  a  concept.  (Many  terms  chosen  for  one  concept  is  a  high  level 
of  specificity.) 

TERM  WEIGHTING:  A  notation  (an  asterisk)  placed  adjacent  indexing 
terms  by  an  indexer  to  indicate  the  greater  importance  Via1  +hat 

term  has. 
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Figure  D-9 

RETRIEVAL  DATA:  Question  223A 


*  Report*  from  Document  Set  H 

Figure  D-10 

RETRIEVAL  DATA:  Question  133 


Figure  D-12 


RETRIEVAL  DATA  FOR  SOURCE  DOCUMENT  QUESTIONS 


(x  Retrieval) 


Figure  D-13 

LIST  OF  TEST  QUESTIONS 


57.  What  types  of  available  anov  and  ice  removal 
equipment  are  suitable  for  use  on  airport  runways, 
tariways,  aprons,  etc? 

85.  Wbat  methods  are  available  for  determining  the 
reliability  of  semiconductors? 

88.  What  is  the  current  state-of-the-art  with  respect 
to  measuring  the  reliability  of  electronic  components, 
particularly  transistors,  capacitors,  relays,  and 
resistors? 

109.  What  are  the  hazards  to  aircraft  colliding  with 
other  objects  in  flight,  such  as  birds,  weather  bal¬ 
loons,  etc?  Fixed  objects  (water  towers,  power  lines, 
antennas,  etc.)  and  other  aircraft  are  not  to  be 
considered. 

111.  What  effect  will  the  introduction  of  supersonic 
transports  into  the  civil  air  fleet  have  on  the  cost 
of  the  air  traffic  control  system  (i.e.,  cost  of 
retraining,  new  navigation  aids,  more  personnel,  etc.)? 

133.  What  methods  are  available  for  forecasting 
thunderstorms  by  passive  detection?  What  spectral 
data  is  available  on  the  emissions  (if  any)  from 
thunderstorm  clouds? 

162.  What  are  the  technical  and  social  problems 
imposed  by  jet  engine  noise  (not  sonic  booms), 
including  methods  for  reducing  or  suppressing  it? 

205.  What  methods  are  available  for  the  measurement 
and  detection  of  high-altitude  clear  air  turbulence? 
What  is  the  frequency,  geographical,  and  altitude 
distribution  of  clear  air  turbulence? 

223A.  What  accident  hazards  do  birds  present  to 
aircraft  during  take-off  and  landing? 

223B.  What  methods  ("scare  devices")  are  available 
for  ridding  terminal  areas  of  birds? 

270.  What  information  is  available  on  airport  land  use 
planning  with  regard  to  aircraft  noise? 
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Figure  D-14 

\  LIST  OF  SOURCE  DOCUMENT  QUESTIONS 

t 

1  1.  What  materials  can  be  added  to  fuels  to  prevent  icing  ? 

2.  Identify  tests  of  radio  transmitter-receiver 8. 

3.  Find  reports  with  data  on  the  lift  characteristics  of  double-slotted-flaps  on 
swept -oaok  wings . 

{  • 

i 

4.  What  has  been  done  on  the  use  of  millimeter  wave  superheterodyne  receivers 
in  radiometers  ? 

5.  What  information  is  avails  hie  on  the  use  of  binary  coding  in  digital  communi¬ 
cations  systems  ? 

6.  Need  information  on  performance  of  traveling  wave  antennas,  particularly 
the  gain  limitations  and  phase  modulations . 

7.  What  effect  does  the  use  of  mid-chord  flaps  have  on  take-off  performance  ? 

8.  What  has  been  done  to  develop  automated  graph  reading  for  computer  input? 

9.  What  chamicals  are  available  for  removing  ice  from  pavements  and  runways  ? 

10.  What  are  some  of  the  disturbances  and  annoyances  of  aircraft  and  airports 
that  cause  communities  to  oppose  their  location  ? 

11.  What  studies  exist  for  detecting  aircraft  in  the  terminal  area,  that  is,  air¬ 
port  surface  traffic  with  radio -doppler  sensors  ? 

12.  Need  a  functional  description  of  the  facilities  needed  for  data  processing 
systems  that  can  be  used  for  ATC. 

13.  What  are  eome  of  the  requirements  of  ATC  display  that  include  weather 
communication  ? 

14.  Identify  studies  that  establish  the  limits  for  "degrees  of  smoothness"  beyond 
which  a  runway  or  taxiway  is  defined  as  rough. 

15.  What  methods  are  available  for  predicting  the  output  signal -to-noise  ratio 
of  an  amplitude  modulated  radio  receiver  ? 

16.  What  is  the  effect  of  training  on  the  performance  of  electronics  maintenance 
personnel  in  aviation  ? 

17.  Want  detailed  information  on  bright  displays,  particularly  circuits  for  alpha 
numeric  data  and  symbolic  data. 
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18.  Find  information  on  the  reduction  of  flight  plan  into  optimum  flight  head¬ 
ings  and  altitudes . 

19.  What  is  the  accuracy  of  position  reporting  under  IFR  ? 

20.  How  can  altitude  be  coded  and  read  in  on  air  traffic  control  system? 
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APPENDIX  E 

DESCRIPTORS  ADDED  TO  THE  THESAURUS 


ABRASIVES 

(Material*  (Application)) 

AERIAL  PICKUP  SYSTEMS 

(Aircraft  Equipment) 

AIR-DROP  OPERATIONS 

(Military  Operations) 

AIR  FORCE  LOGISTICS  COMMAND 
(Military  Organliatlons) 

AIR  SCOOPS 

(Engines  and  Engine 
Operations) 

AIRCRAFT  INTERCEPTION 

(Detection  and  Tracking) 

ALGEBRA 

(Mathematics) 

ALUMINUM  COMPOUNDS 

(Chemical  Compounds) 

AMMETERS 

(Electrical  and  Electronic 
Measurement) 

ANGLE  OF  ARRIVAL 

(Electromagnetic  Wave 
Phenomena) 

ANODES  (ELECTROLYTIC  CELL) 

(Electric  Power  Sources) 

ANTIFOGGING  AGNETS 

(Materials  (Application)) 

ARC-WELDING 

(Metal  Joining) 

ARMAMENT 

(Warfare  and  Weapons) 


♦Non-ASTIA  descriptors 


ARMY  EQUIPMENT 
(Logistics) 

ARSENIDES 

(Chemical  Compounds) 

ARTIFICIAL  INTELLIGENCE 
(Bionics) 

ATMOSPHERE  MODELS 
(Models) 

ATMOSPHERIC  TIDES 

(Meteorology  and  Clima¬ 
tology) 

ATTACK  BOMBERS 
(Aircraft) 

AUTOGYRO  ROTORS 

(Aerodynamic  Configurations) 

♦AUTOMATIC  FREQUENCY  CONTROL 
(Electrical  and 
Electronics  Equipment) 

♦AUTOMATIC  LANDING  SYSTEMS 
(Aeronautics) 

♦AWARDS 

(General  Services  and 
Supplies) 

BACKGROUND 

(Abstract  Concepts) 

BACKWARD-WAVE  TUBES 
(Electron  Tubes) 

BARIUM  COMPOUNDS 

(Chemical  Compounds) 

BAROMETERS 

(Meteorological  Aids) 
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BBSSKL  FUNCTIONS 

(Mathematics) 

BLEED  SYSTEMS 

(Hydraulic  and  Pneuaat  lc 
Systems) 

BLOOD  CHEMISTRY 

(Biochemistry) 

♦BOMB  DETECTION 

(Safety  and  Accidenta) 

BORANES 

(Chealcal  Compounds) 

BORON 

(Chaalcal  Compounds) 

BORON  COMPOUNDS 

(Chaalcal  Elcaanta) 

BURNS 

(Wounds  and  Injuries) 
CADMIUM 

(Chealcal  Elements) 
CALCIUM 

(Chealcal  Eleaants) 

CAPACITANCE  BRIDGES 

(Electrical  and  Electronic 
Kqulpawnt) 

CAPTIVE  TESTS 

(Laboratories  and  Test 
Facilities) 

CARBON 

(Chealcal  Elements) 

CARBON  ARC  LAMPS 

(Instruaentat Ion) 

CARBON  BUCK 

(Materials) 

CARRIERS  (RADIO  WAVES) 

(Radio) 


CATHODE  FOLLOWERS 

(Electrical  and  Electronic 
Circuits) 

CATHODES 

(Electrical  and  Electronic 
Equipment) 

CELLULOSIC  FUSTICS 
(Plastics) 

CENTRAL  NERVOUS  SYSTEM 
(Anatoay) 

CERAMIC  COATINGS 

(Finishes  and  Finishing) 

CHEMICAL  MILLING 

(Industrial  and  Laboratory 
Processes) 

CHILDREN 

(Personnel) 

CHROMIUM 

(Chealcal  Eleaents) 

CIVIL  ENGINEERING 

(Research  Fields) 

♦CLEAR  AIR  TURBULENCE 

(Meteorology  and  Clima¬ 
tology) 

♦CLEARANCE 

(Aeronautics) 

COHERENT  RADAR 
(Radar) 

COMBUSTION  PRODUCTS 
(Combustion) 

COMPRESSOR  NOISE 
(Acoustics) 

CONDUCTIVITY 

(Physical  and  Physico¬ 
chemical  Concepts) 

♦CONFERENCES 

(Document at  ion) 
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CONVERGENT -DIVERGENT  NOZZLES 
(Rockets) 

CONVEX  SETS 

(Mathematics) 

COOLANTS 

(Materials  (Application)) 

CORNEA 

(Anatomy) 

★COUPLERS 

(Electrical  and  Electronic 
Equipment) 

★CRASH  TESTING 

(Laboratories  and  Test 
Facilities) 

CRIMINOLOGY 

(Social  Sciences) 

★CRUISING 

(Aeronautics) 

CRYSTAL  OSCILLATORS 

(Electrical  and  Electronic 
Equipment) 

CUMULUS  CLOUDS 

(Meteorology  and  Clima¬ 
tology) 

DACRON 

(Textiles  and  Fibers) 
DESICCANTS 

(Materials  (Application)) 

DISCONNECT  FITTINGS 

(Couplings,  Fittings  and 
Fastenings) 

DISPERSION  HARDENING 

(Industrial  and  Laboratory 
Processes) 

DRIFTMETERS 

(Flight  Instruments) 


DUAL-ROTATION  PROPELLERS 
(Propulsion) 

ELECTRIC  PROPULSION 
(Propulsion) 

ELECTROLYTIC  CAPACITORS 

(Electrical  and  Electronic 
Equipment) 

ELECTRON  BOMBARDMENT 

(Particle  Accelerators) 

ELECTRON  DENSITY 

(Meteorology  and  Clima¬ 
tology) 

ELECTRON  LENSES 

(Electron  Tubes) 

ELECTR00PT1CAL  PHOTOGRAPHY 
(Photography) 

ELLIPSOIDS 

(Geometric  Forms) 

EMBRYOS 

(Physiology) 

EMPLOYEE  RELATIONS 

(Social  Sciences) 

ENGINE  SURGE 

(Engines  and  Engine 
Operations) 

ENGINEERING  GEOLOGY 

(Geology  and  Seismology) 

EPITAXIAL  GROWTH 

(Crystallography) 

EUTECTICS 

(Materials  (Physical 
State)) 

EXCITATION 

(Atomic  and  Molecular 
Physics) 
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EXPLODING  HIRES 

(Aaeunltion  and  Expletives) 

EXPLOSIVES  INITIATORS 

(Aramltlon  and  Expletives) 

FEDERAL  BUDGETS 
(Economic*) 

FIBRIN 

(ProCtlnt) 

FIELD  THEORY 

(Electricity  and  Magna  tin) 

FILM  PROJECTORS 

(Photography) 

FILM  READERS 

(Photography) 

FILTERS  (ELECTROMAGNETIC  HAVE) 
(Filters) 

FLAME  HOLDERS 

CCoabust Ion) 

FLEXIBLE  STRUCTURES 

(Structural  Engineering) 

FOREIGN  POLICY 

(Social  Sciences) 

FOULING 

(Abstract  Concepts) 

FOUNDATIONS  (STRUCTURES) 

(Structural  Engineering) 

FREE-FLIGHT  TRAJECTORIES 
(Mechanics) 

FREEZING 

(Ihyslcal  and  Physlco- 
chealcal  Concepts 


FUEL  TRUCES 

(Vehicles) 

GAS  GENERATOR  ENGINES 

(Engines  and  Engine 
Operations) 

GROWTH 

(Physiology) 

GUIDED  MISSILE  COMPONENTS 
(Guided  Missiles) 

GUIDED  MISSILE  SIMULATORS 
(Guided  Missiles) 

GUN- LAUNCHED 

(Modifiers) 

«HALABY  SPEECHES 

HANGARS 

(Structural  Engineering) 

HARMONIC  OSCILLATORS 

(Electrical  and  Electronic 
Equipment) 

HEAT  OF  FUSION 

(Theraodynaalcs ) 

HEAT  TRANSFER  COEFFICIENTS 
(Theraodynaalcs) 

HEMORRHAGE 

(Pathology) 

HIBERNATION 

(Biology) 

HORIZON  SCANNERS 

(Flight  Control  Systcas) 

HORN  ANTENNAS 
(Antennas) 


FUEL  METERS 

(Engines  and  Engine 
Operations) 


HYDRAULIC  COUPLINGS 

(Couplings,  Fittings 
and  Fasteners) 


FUEL  THICKENERS 

(Materials  (Application)) 
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HYDRIDES 

(Chemical  Compounds) 

HYDROLYSIS 

(Chemical  Reactions) 


IMAGE  INTENSIFIERS  (ELECTRONICS) 
(Electron  Tubes) 

IMPEDANCE  BRIDGES 

(Electrical  and  Electronic 
Equipment) 

INCENDIARY  PROJECTILES 

(Warfare  and  Weapons) 


INDUSTRIAL  PSYCHOLOGY 

(Psychology  and  Psycho¬ 
metrics) 


INFRARED  SPECTROPHOTOMETERS 
(Spectroscopy) 

INGESTION  (ENGINES) 

(Engines  and  Engine 
Operations) 


IRASERS 

(Amplifiers) 

IRON  COMPOUNDS 

(Chemical  Compounds) 

ISOCYANATE  PLASTICS 
(Plastics) 

LABOR 

(Social  Sciences) 

LABOR  UNIONS 

(Social  Sciences) 

LAUNCH  VEHICLES  (AEROSPACE) 
(Rockets) 

HEADING  EDGE 

(Aerodynamic  Configur¬ 
ations) 

LOW-TEMPERATURE  LUBRICANTS 
(Lubrication  and 
Bearing*.*) 

LOW-TEMPERATURE  RESEARCH 
(Research  Fields) 

LYMPHOCYTES 

(Hematology) 

MACHINE  TRANSLATION 
(Documentation) 

MAGNETIC  CORE  STORAGE 

(Cosrputers  and  Data 
Systems) 

MAGNETIC  GUIDANCE 

(Navigation  and  Guidance) 

MAGNETIC  STORMS 

(Meteorology  and  Clima¬ 
tology) 

MAGNETOMETERS 

(Instrumentation) 


INTEGRATION 

(Metheaiatlcs) 

INTESTINES 

(Anatomy) 


HYDROPHONES 

(Acoustics) 

^HYDROPLANING 

(Fluid  Dynamics) 


INDUSTRIAL  PROCUREMENT 
(Logistics) 


INDUSTRIAL  RELATIONS 
(Social  Sciences) 

INDUSTRIAL  TRAINING 
(Training) 

INFECTIONS 

(Pathology) 
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*MAN-MACHINE  SYSTEMS 

MUTATIONS 

j  - 

(Abstract  Concepts) 

(Biology) 

i 

i 

MARKERS 

NAVAL  EQUIPMENT 

(Pyrotechnics) 

(Logistics) 

1 

• 

MARTENSITE 

NAVAL  RESEARCH  LABORATORIES 

(Metallurgy  and  Metal lo- 

(Laboratories  end  Test 

graphy) 

Facilities) 

MATCHED  FILTERS 

NAVY 

(Filters) 

(Military  Organization) 

MATERIAL  CONTROL 

NEGATIVE  RESISTANCE  CIRCUITS 

(Logistics) 

(Electrical  end  Electronic 

Equipment) 

*MC  XEE 

NEWSPAPERS 

MEMWANES 

(Documentation) 

(Material  (Physical  State)) 

NIGHT  LANDINGS 

| 

METALLIC  CRYSTALS 

(Aeronaut  ice) 

i 

i 

i 

(Crystallography) 

NOCTILUCENT  CLOUDS 

MICROAFALYSIS 

^Meteorology  and  Cl las- 

i 

.  i 

(Chemistry) 

tology) 

i 

MICROFILM 

NOZZ1X  CLUSTERS 

» 

(Photography) 

(Rockets) 

i 

MILITARY  PUBLICATIONS 

NOZZLE  INSERTS 

! 

(Document at  Ion) 

(Rockets) 

MILITARY  STRATEGY 

NUCLEAR  SPINS 

T 

(Military  Operations) 

(Nuclear  Physics) 

MINING  ENGINEERING 

OBESITY 

(Research  Ftelds'V 

(Physiology) 

MIXTURES 

OFFICE  EQUIPMENT  AND  SUPPLIES 

(Materials  (niyslcal  State)) 

(General  Services  and 

• 

MOBILISATION 

Suppllea) 

(I oglst les) 

OPERATORS  (MATHEMATICS) 

* 

• 

MONEY 

(Mathematics) 

(Economics) 

OPTIMIZATION 

(Mathematics) 

MULTIPLE  OPERATION 

i 

(Abstract  Concepts) 

\ 

\ 

' 
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PARAMAGNETIC  RESONANCE 

(Electricity  and  Magnetics) 

PARENTERAL  INFUSIONS 
(Medicine) 

PARTICLE  SIZE 

(Physical  and  Physico- 
chealcal  Concepts) 

PATENTS 

(Doc  mentation) 

♦PEAK  AIR  TRAFFIC 
(Aeronautics) 

PENTABORANES 

(Chemical  Compounds) 

PERMEABILITY 

(Physical  and  Physico¬ 
chemical  Concepts) 

PHASE  STUDIES 

(Physical  and  Physico¬ 
chemical  Concepts) 

PHOSPHORESCENCE 

(Optics) 

PHOSPHORUS  TRANSFERASES 
(Enzymes) 

PHOTOGRAPHIC  FILM 
(Photography) 

PHOTOGRAPHIC  FILTERS 
(Photography) 

PHOTOGRAPHIC  INTELLIGENCE 
(Intelligence) 

PIEZOELECTRIC  TRANSDUCERS 
(Instrumentation) 

PITCH  DISCRIMINATION 
(Acouat lcs) 

PLUG  NOZZLES 
(Rockets) 


PNEUMATIC  BRAKES 
(Vehlbles) 

POLITICAL  SCIENCE 

(Social  Sciences) 

POLYNOMIALS 

(Mathematic*) 

POROSITY 

(Physical  and  Riyslco- 
chemlcal  Concept*) 

POROUS  MATERIALS 
(Materials) 

POTASSIUM  ALLOYS 
(Alloys) 

POTENTIOMETERS 

(Electrical  and  Electronic 
Equipment)  ' 

PREGNANCY 

(Phyalology) 

PRINTED  CIRCUITS 

(Electrical  and  Electronic 
Equipment) 

PROPELLER  NOISE 
(Acouat lea) 

PULSE  COUNTERS 

(Electrical  and  Electronic 
Equipment) 

PULSE  DISCRIMINATORS 

(Electrical  and  Electronic 
Equipment) 

QUANTUM  MECHANICS 

(Research  Fields) 

RADAR  CONFUSION  REFLECTORS 

(Electromagnetic  Warfare) 

RADAR  INTERCEPTION 

(Electromagnetic  Warfare) 
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RADIO  HOMING 

(Navigation  and  Guidance) 

RADIOLOGICAL  WARFARE 
(Warfare) 

RAMJET  ENGINE  NOZZLES 

(Engine*  and  Engine 
Operations) 

RAMJET  inlets 

(Engines  and  Engine 
Operations) 

RARE  EARTH  COMPOUNDS 

(Chemical  Compounds) 

RARE  EARTHS 

(Chemical  Compounds) 

RECUIT]  iC 

(Military  Operations) 

REFRACTORY  COATDJGS 

(Finishes  and  Finishing) 

ROCKET  PROPELLANTS 

(FW»hes  and  Finishing) 

ROTARY  SWITCHES 

(Electrical  and  *;<•.  ctvonic 
Equipment) 

♦ROTOR  HUBS 

(Aircraft  Structures) 

rotorchutes 

(Aeronaut ics) 

SATELLITE  ATTITUDE 

(Space  Technology) 

SEALS  (STOPPERS) 

(Adhesive  and  Se*Js) 

SELENIDES 

(Chemical  Compounds) 

SELF-SEALING  COUPLINGS 

(Couplings ,  Fittings 
and  Fast entngs) 


SEMICIRCULAR  CASALS 
(Anatomy) 

♦SENSING  ELEMENTS 

(Inatrumentatlon) 

SEQUENCES 

(Mathematics) 

SHIPS 

(Shipa  and  Boats) 

SILICIC  ACIDS 

(Chemical  Compounds) 

SILT 

(Geology  ar.d  Seismology) 

SINGLE  SIDEBAND  COMMUNICATIONS 
SYSTEMS 

(Communications  Systems) 
SINTERING 

(Industrial  and  Laboratory 
Processes) 

SLURRY  FUELS 
(Fuels) 

♦SLUSH 

(Meteorology  aad  Clima¬ 
tology) 

SOLUTIONS 

(Materials  (Physical  State)) 

SOUND  RANGING 

(Detection  and  Tracking) 

SPACE  COMMUNICATIONS  SYSTEMS 
(Ootmaunications  Systems) 

SPACE  NAVIGATION 

(Navigation  and  Guidance) 

♦SPACE  TECHNOLOGY 

(Space  Technology) 

SPaCEBOANE 

(Modi tiers) 

3PARY  IGNITION 

(Combustion) 
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SPECIFIC  IMPULSE 

(Rocket  Propellents) 

SPEECHES 

(Document at' on) 

SPINNERS 

(Aerodynamic  Con figuret ions) 

SPRAY  NOZZLES 

(Fluid  Dynamics) 

STIFFENED  CYLINDERS 

(Ceosetrlc  Forms) 

STRAIN  MECHANICS 

(Mechanical  Properties) 

STRATUS  CLOUDS 

(Meteorology  and  Clime. 

toiogy) 


SUBMARINE  PERSONNEL 
(Personnel) 

SUCROSE 

(Carboh  yd  rat  es ) 

SUNSPOTS 

(Astrononv) 

SWEPT -FORWARD  WINGS 

(Aerodynamic  Configurations) 

SYNTHETIC  STONES 
(Mineralogy) 

SYSTEMS  ENGINEERING 

(Research  Fields) 

TAIL  HELICCOPTOR  ROTORS 

(Aerodynamic  Configurations) 

TAILLESS  AIRPLANES 
(Aircraft) 

TARGET  ANGLE 

(Fire  Control  and  Bombing) 

TAYLOR'S  SERIES 

(Mathematics) 


^TECHNIQUES 

(Abstract  Concept*) 

telegraph  equipment 

(Telephone,  Telegraph 
and  Teletype) 

b«)  TELESCOPES 

(Optical  Equipment) 

television  ANTENNAS 

(Antennas) 

TELEVISION  CONVERTERS 
(Television) 

TEST  CONSTRUCTION  (PSYCHOLOGY) 
(Psychology  end  Psycho, 
aetrica) 

TETRODES 

(Electron  Tubes) 

TEXTBOOKS 

(Documentation) 

THERMIONIC  CONVERTERS 

(Electric  Pover  Sources) 

thermoelectricity 

(Electricity  and  Magnetism) 

)  THORIUM  ALLOYS 
(Alloys) 

TITANIUM  COMPOUNDS 

(Chemical  Compounds) 

TOGGLE  switches 

(Electrical  and  Electronic 
Equipment) 

i 

TOUGHNESS 

(Mechanical  Properties) 

TOXINS  &  ANTITOXINS 
(Pharmacology) 

TRANSITION  TEMPERATURE 

(Physical  and  Physico¬ 
chemical  Concepts) 


TRANSPORT  PROPERTIES 

(Physical  and  Ihysico- 
cheaical  Concepts) 

TROPOPAUSE 

(Meteorology  and  Cliaa- 
tology) 

TUNGSTEN  ALLOYS 
(Alloys) 

TUNING  DEVICES 

(Electrical  and  Electronic 
Equipment) 

TWILIGHT 

(Meteorology  and  Clima¬ 
tology) 

UNDERGROUND  EXPLOSIONS 
(Explosions) 

UNDERGROUND  STRUCTURES 

(Structural  Engineering) 

UNIVERSITIES 

(Social  Sciences) 

VACUUM  APPARATUS 

(Instrumental  on) 

VACUUM  SEALS 

(Adhesives  and  Seals) 

VAPOR  PRESSURF 

(Physical  and  Physico¬ 
chemical  Concepts) 

VARIABLE -INCIDENCE  WINGS 

(Aerodynamic  Configurations) 

VARIABLE- SWEEP  WINGS 

(Ai . ^dynamic  Configurations) 

VECTOR  ANALYSIS 

(Mathemat lcs) 

VESTIBULAR  APPARATUS 
(Anatomy) 


VIABILITY 

(Physiology) 

VOLTMETERS 

(Electrical  and  Electronic 
Measurement) 

VORTEX  THERMOMETERS 
(Instrumentation) 

WALKIE-TALKIES 

(Radio) 

WASTE  GASES 

(Sanitary  Engineering) 

WATER  INJECTION 

(Engines  and  Engine 
Operat ion) 

WATER  SUPPLIES 

(Oceanography  and  Hyrology) 

WATTMETERS 

(Electrical  and  Electronic 
Equipment ) 

WAVE  ANALYZERS 

(Instrument at  Ion) 

WAVEGUIDE  CIRCULATORS 

(Electrical  and  Electronic 
Equipment ) 

WEAPONS 

(Warfare  and  Weapons) 

ZINC  COMPOUNDS 

(Chemical  Compounds) 
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DESCRIPTORS  DELETED  FROM  THE  THESAURUS 


# 


ABSORPTION  BIOLOGICAL 

ALKALI  METAL  ALLOYS 

ALKALI  METAL  COMPOUNDS 

ALKALINE  EARTH  COMPOUNDS 

ALKALINE  EARTH  METALS 

ALKALOIDS 

ANTHELMINTICS 

ANTICONVULSANTS 

ANTIMALARIALS 

ANTISONAR  COATINGS 

ARMOR 

ARTHROPODS 

AZIDES 

BARBITURATES 

BASES  (CHEMISTRY) 

BILIARY  SYSTEM 
BIOLOGICAL  PRODUCTS 
BIOSYNTHESIS 
BLOOD  GROUPS 
BONE  MARROW 
CARBONATE  Ml  K  SR  AT.  S 
CARDIOACTIVE  AGENTS 
CHALCOGENS 
CHARGES  (EXPLOSIVE) 


CHECK  VALVES 

CHEMICAL  EQUILIBRIUM 

CHLOR PROMAZINE 

CHROMOPROTEINS 

CINCHONA  ALKALOIDS 

COMPRESSOR  PARTS 

CONTROLLABLE-THRUST  ROCKET 
MOTORS 

CRYPTOGRAPHY 

CULTURE 

CUTTING  TOOLS 

CYANATES 

DECEPTION 

DIAPHRAGMS  (MECHANICS) 

DI STANCE -TO-GO  MARKERS 
DUAL-THRUST  ROCKET  MOTORS 
ELECTRIC  BRIDGES 
ELECTROSTATIC  GENERATORS 
EMBOLISM 

EMBRYONATED  EGG  TECHNIQUE 
END  ORGANS 

ENGINE  AIR  SYSTEMS  COMPONENTS 
ENGINE  CLUSTERS 
EUBACTERIALES 
FAULTS  (GEOLOGY) 
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FIBERS  (  NATURAL) 

FIBRIN 
FIELD  WIRE 

FIRE  CONTROL  COMPUTERS 

FIRE  CONTROL  SYSTEM  COMPONENTS 

FIRING  TESTS  (ORDNANCE) 

FLUORESCENT  SCREENS 

FLUXMETERS 

FOLDS  (GEOLOGY) 

FOOD  DISPENSING 
FUZE  FUNCTIONING  ELEMENTS 
GAS  FILTERS 
GEMS  (MINERALS) 

GLASS  SEALS 

GROUP  VIII  ELEMENTS 
GUIDED  MISSILE  MODELS 
HALIDES 

HEALTH  PHYSICS  INSTRUMENT AT I OM 

HEAT  ENGINES 

HEMOPOIETIC  SYSTEM 

HEXOSES 

HYDRIDES 

IMAGE  MOTION  COMPENSATION 
INDUSTRIAL  EQUIPMENT 


INFRARED  PHENOMENA 

INTEGUMENTARY  SYSTEM 

IN1ERNAL  COMBUSTION  ENGINE 
NOISE 

ISOTOPES 

JOURNAL  BEARINGS 
LACTOBACILLACEAE 
LIFE  SPAN 

LINEAR  ACCELERATORS 

LIQUID  PILTERS 

LIQUID  LEVEL  CONTROL 

LITHOSPHERE 

MAGNUS  FORCE 

MARINE  SAFETY  EQUIPMENT 

MARKERS 

MATERIAL  REMOVAL 
MATERIAL  SEPARATION 
METALLOID  ALLOYS 
METALLOIDS 
MICROPALEONTOLOGY 
MILITARY  TACTICS 
MINERALS 

MORPHOLOGY  (BIOLOGY) 
MOSAICS (LIGHT  SENSITIVE) 
MYDRIATICS 

NAVAL  SHORE  ESTABLISHMENTS 


i 

* 

* 

i 
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NAVAL  VESSELS  (COMBATANT) 
NUCLEAR  PARTICLES 
NUCLEAR  PROPULSION 
ORES  (NONMETALLIC) 

ORGANIC  PIGMENTS 
PALEONTOLOGY 

PARTICLE  ACCELERATOR  COMPONENTS 

PARTICLE  ACCELERATOR  TECHNIQUES 

PARTICLE  ACCELERATORS 

PARTICLE  BEAMS 

PERMEABILITY 

PHENOTHIAZINES 

PHOSPHATES 

PHOTOCATHODES 

PHOTOGRAPHIC  RECONNAISSANCE 

PHOTOGRAPHIC  RECORDING  MEDIA 

PIRICULARIA 

POINT- INITIATING  FUZES 

POLYAMIDE  PLASTICS 

POWER  PLANTS  (ESTABLISHMENTS) 

PROPELLING  CHARGES 

PSYCHOANALEPTIC  AGENTS 

PSYCHOTROPIC  AGENTS 

PUBLIC  HEALTH 

PURSUIT  COURSES 


QUINOLINE  ALKALOIDS 
REACTOR  LATTICE  PARAMETERS 
REACTOR  SYSTEM  COMPONENTS 
REPRODUCTIVE  SYSTEM 
RETICUL-END-THELIAL  SYSTEM 
ROCK  FORMING  MINERALS 
ROCKET  COMPONENTS 
SANDSTONE 
SEAFOOD 

SEDIMENTARY  ROCK 
SEEDS 

SENSE  ORGANS 
SEWAGE 

SHIP  AUXILIARY  EQUIPMENT 
SMALL  TOOLS 
SOCIAL  SCIENCES 
SOLAR  ATMOSPHERE 
SOLID  ROCKET  FUELS 
SPACE  PROBES 
STRATEGIC  WARFARE 
STRATEGIC  WEAPONS 
STRUCTURAL  GEOLOGY 
SUGAR  ACIDS 
SURFACE -TO-UNDERWATER 
TACTICAL  WEAPONS 


118 


TAR 

TEMPERATURE  COETFICIENT  OF 
REACTIVITY 

THERMOPLASTICS 

THYRATRONS 

TRANSPORTER-ERECTORS 

TUNG  OIL 

TURBINE  PARTS 

ULTRAVIOLET  OPTICAL  MATERIALS 
ULTRAVIOLET  RECEIVERS 
UNDERWATER -TO- UNDERWATER 
UNDERWATER  TRACKING 
UREIDES 

URINARY  SYSTEM 

URONIC  ACIDS 

VEHICLE  ACCESSORIES 

VEHICLE  CHASSIS  COMPONENTS 

VETERINARY  MEDICINE 

VITAMINS 

WAXES 

WHITE  PHOSPHORUS 
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